Issue
I created an Iot Edge solution using Visual Studio Code via View -> Command Palette... -> AzureIotEdge:NewIoTEdgeSolution using the C# Module Module Template. This generates an Iot Edge solution with two Modules: tempSensor creates messages with dummy telemetry-data and SampleModule is supposed to receive those messages, print them and then pass them on to IotHub.
I build the solution for arm32 and deployed it to a Rasperry Pi (running latest Raspbian). The modules are then displayed as running on the device but if I monitor d2c-messages there are none arriving in IoT Hub.
If I run sudo iotedge logs tempSensor -f on the device, I can see messages are being generated.
If I run sudo iotedge logs SampleModule -f I only see IoT Hub module client initialized. Which is logged in the Init() Method. There is no logging from the PipeMessage Method which seems not to be reached at all.
If I build the solution for amd64 and deploy it to a virtual machine, everything is working as expected.
Is there a way to get it working on a Raspberry Pi running Raspian?
By the way: I tried the Azure Functions C# Module Template as well. Same result: On amd64 it works but on arm32 it does not.
UPDATE: Here is the logging of edgeAgent:
█████╗ ███████╗██╗ ██╗██████╗ ███████╗
██╔══██╗╚══███╔╝██║ ██║██╔══██╗██╔════╝
███████║ ███╔╝ ██║ ██║██████╔╝█████╗
██╔══██║ ███╔╝ ██║ ██║██╔══██╗██╔══╝
██║ ██║███████╗╚██████╔╝██║ ██║███████╗
╚═╝ ╚═╝╚══════╝ ╚═════╝ ╚═╝ ╚═╝╚══════╝
██╗ ██████╗ ████████╗ ███████╗██████╗ ██████╗ ███████╗
██║██╔═══██╗╚══██╔══╝ ██╔════╝██╔══██╗██╔════╝ ██╔════╝
██║██║ ██║ ██║ █████╗ ██║ ██║██║ ███╗█████╗
██║██║ ██║ ██║ ██╔══╝ ██║ ██║██║ ██║██╔══╝
██║╚██████╔╝ ██║ ███████╗██████╔╝╚██████╔╝███████╗
╚═╝ ╚═════╝ ╚═╝ ╚══════╝╚═════╝ ╚═════╝ ╚══════╝
2018-12-21 07:08:27.323 +00:00 [INF] - Edge agent attempting to connect to IoT Hub via Amqp_Tcp_Only...
2018-12-21 07:08:31.986 +00:00 [INF] - Created persistent store at /tmp/edgeAgent
2018-12-21 07:08:34.193 +00:00 [INF] - Edge agent connected to IoT Hub via Amqp_Tcp_Only.
2018-12-21 07:08:37.624 +00:00 [INF] - Plan execution started for deployment 8
2018-12-21 07:08:37.654 +00:00 [INF] - Executing command: "Stop module SampleModule"
2018-12-21 07:08:37.766 +00:00 [INF] - Executing command: "Remove module SampleModule"
2018-12-21 07:08:37.964 +00:00 [INF] - Executing command: "Saving SampleModule to store"
2018-12-21 07:08:38.141 +00:00 [INF] - Executing command: "Command Group: (
[Create module SampleModule2]
[Start module SampleModule2]
)"
2018-12-21 07:08:38.152 +00:00 [INF] - Executing command: "Create module SampleModule2"
2018-12-21 07:08:42.678 +00:00 [INF] - Executing command: "Start module SampleModule2"
2018-12-21 07:08:45.749 +00:00 [INF] - Executing command: "Command Group: (
[Stop module tempSensor]
[Start module tempSensor]
[Saving tempSensor to store]
)"
2018-12-21 07:08:45.750 +00:00 [INF] - Executing command: "Stop module tempSensor"
2018-12-21 07:08:45.759 +00:00 [INF] - Executing command: "Start module tempSensor"
2018-12-21 07:08:48.423 +00:00 [INF] - Executing command: "Saving tempSensor to store"
2018-12-21 07:08:48.491 +00:00 [INF] - Executing command: "Command Group: (
[Stop module edgeHub]
[Start module edgeHub]
[Saving edgeHub to store]
)"
2018-12-21 07:08:48.491 +00:00 [INF] - Executing command: "Stop module edgeHub"
2018-12-21 07:08:48.501 +00:00 [INF] - Executing command: "Start module edgeHub"
2018-12-21 07:08:51.895 +00:00 [INF] - Executing command: "Saving edgeHub to store"
2018-12-21 07:08:51.900 +00:00 [INF] - Plan execution ended for deployment 8
2018-12-21 07:08:52.743 +00:00 [INF] - Updated reported properties
2018-12-21 07:08:57.811 +00:00 [INF] - HealthRestartPlanner is clearing restart stats for module 'tempSensor' as it has been running healthy for 00:10:00.
2018-12-21 07:08:57.814 +00:00 [INF] - HealthRestartPlanner is clearing restart stats for module 'edgeHub' as it has been running healthy for 00:10:00.
2018-12-21 07:08:57.815 +00:00 [INF] - Plan execution started for deployment 8
2018-12-21 07:08:57.818 +00:00 [INF] - Executing command: "Saving tempSensor to store"
2018-12-21 07:08:57.819 +00:00 [INF] - Executing command: "Saving edgeHub to store"
2018-12-21 07:08:57.820 +00:00 [INF] - Plan execution ended for deployment 8
2018-12-21 07:08:58.038 +00:00 [INF] - Updated reported properties
2018-12-21 07:09:03.239 +00:00 [INF] - Updated reported properties
And here some of the logging of edgeHub (there is lots more but seems to be all the same Exceptions):
2018-12-21 07:28:23.450 +00:00 [WRN] - Error sending messages to module bewaPi1/SampleModule2
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at DotNetty.Buffers.HeapArena.NewChunk(Int32 pageSize, Int32 maxOrder, Int32 pageShifts, Int32 chunkSize)
at DotNetty.Buffers.PoolArena`1.AllocateNormal(PooledByteBuffer`1 buf, Int32 reqCapacity, Int32 normCapacity)
at DotNetty.Buffers.PoolArena`1.Allocate(PoolThreadCache`1 cache, PooledByteBuffer`1 buf, Int32 reqCapacity)
at DotNetty.Buffers.PooledByteBufferAllocator.NewHeapBuffer(Int32 initialCapacity, Int32 maxCapacity)
at DotNetty.Buffers.AbstractByteBufferAllocator.Buffer(Int32 initialCapacity, Int32 maxCapacity)
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.ByteBufferConverter.ToByteBuffer(Byte[] bytes) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/ByteBufferConverter.cs:line 39
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.ProtocolGatewayMessageConverter.FromMessage(IMessage message) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/ProtocolGatewayMessageConverter.cs:line 101
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.DeviceProxy.SendMessageAsync(IMessage message, String input) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/DeviceProxy.cs:line 73
at Microsoft.Azure.Devices.Edge.Hub.Core.Device.DeviceMessageHandler.SendMessageAsync(IMessage message, String input) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Core/device/DeviceMessageHandler.cs:line 256
at Microsoft.Azure.Devices.Edge.Hub.Core.Routing.ModuleEndpoint.ModuleMessageProcessor.<>c__DisplayClass5_2.<<ProcessAsync>b__0>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Core/routing/ModuleEndpoint.cs:line 98
2018-12-21 07:28:28.517 +00:00 [WRN] - Non retryable exception occurred while sending message.
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at DotNetty.Buffers.HeapArena.NewChunk(Int32 pageSize, Int32 maxOrder, Int32 pageShifts, Int32 chunkSize)
at DotNetty.Buffers.PoolArena`1.AllocateNormal(PooledByteBuffer`1 buf, Int32 reqCapacity, Int32 normCapacity)
at DotNetty.Buffers.PoolArena`1.Allocate(PoolThreadCache`1 cache, PooledByteBuffer`1 buf, Int32 reqCapacity)
at DotNetty.Buffers.PooledByteBufferAllocator.NewHeapBuffer(Int32 initialCapacity, Int32 maxCapacity)
at DotNetty.Buffers.AbstractByteBufferAllocator.Buffer(Int32 initialCapacity, Int32 maxCapacity)
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.ByteBufferConverter.ToByteBuffer(Byte[] bytes) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/ByteBufferConverter.cs:line 39
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.ProtocolGatewayMessageConverter.FromMessage(IMessage message) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/ProtocolGatewayMessageConverter.cs:line 101
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.DeviceProxy.SendMessageAsync(IMessage message, String input) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/DeviceProxy.cs:line 73
at Microsoft.Azure.Devices.Edge.Hub.Core.Device.DeviceMessageHandler.SendMessageAsync(IMessage message, String input) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Core/device/DeviceMessageHandler.cs:line 256
at Microsoft.Azure.Devices.Edge.Hub.Core.Routing.ModuleEndpoint.ModuleMessageProcessor.<>c__DisplayClass5_2.<<ProcessAsync>b__0>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Core/routing/ModuleEndpoint.cs:line 98
2018-12-21 07:28:28.521 +00:00 [WRN] - Error sending messages to module bewaPi1/SampleModule2
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at DotNetty.Buffers.HeapArena.NewChunk(Int32 pageSize, Int32 maxOrder, Int32 pageShifts, Int32 chunkSize)
at DotNetty.Buffers.PoolArena`1.AllocateNormal(PooledByteBuffer`1 buf, Int32 reqCapacity, Int32 normCapacity)
at DotNetty.Buffers.PoolArena`1.Allocate(PoolThreadCache`1 cache, PooledByteBuffer`1 buf, Int32 reqCapacity)
at DotNetty.Buffers.PooledByteBufferAllocator.NewHeapBuffer(Int32 initialCapacity, Int32 maxCapacity)
at DotNetty.Buffers.AbstractByteBufferAllocator.Buffer(Int32 initialCapacity, Int32 maxCapacity)
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.ByteBufferConverter.ToByteBuffer(Byte[] bytes) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/ByteBufferConverter.cs:line 39
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.ProtocolGatewayMessageConverter.FromMessage(IMessage message) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/ProtocolGatewayMessageConverter.cs:line 101
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.DeviceProxy.SendMessageAsync(IMessage message, String input) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/DeviceProxy.cs:line 73
at Microsoft.Azure.Devices.Edge.Hub.Core.Device.DeviceMessageHandler.SendMessageAsync(IMessage message, String input) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Core/device/DeviceMessageHandler.cs:line 256
at Microsoft.Azure.Devices.Edge.Hub.Core.Routing.ModuleEndpoint.ModuleMessageProcessor.<>c__DisplayClass5_2.<<ProcessAsync>b__0>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Core/routing/ModuleEndpoint.cs:line 98
2018-12-21 07:28:33.666 +00:00 [WRN] - Non retryable exception occurred while sending message.
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at DotNetty.Buffers.HeapArena.NewChunk(Int32 pageSize, Int32 maxOrder, Int32 pageShifts, Int32 chunkSize)
at DotNetty.Buffers.PoolArena`1.AllocateNormal(PooledByteBuffer`1 buf, Int32 reqCapacity, Int32 normCapacity)
at DotNetty.Buffers.PoolArena`1.Allocate(PoolThreadCache`1 cache, PooledByteBuffer`1 buf, Int32 reqCapacity)
at DotNetty.Buffers.PooledByteBufferAllocator.NewHeapBuffer(Int32 initialCapacity, Int32 maxCapacity)
at DotNetty.Buffers.AbstractByteBufferAllocator.Buffer(Int32 initialCapacity, Int32 maxCapacity)
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.ByteBufferConverter.ToByteBuffer(Byte[] bytes) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/ByteBufferConverter.cs:line 39
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.ProtocolGatewayMessageConverter.FromMessage(IMessage message) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/ProtocolGatewayMessageConverter.cs:line 101
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.DeviceProxy.SendMessageAsync(IMessage message, String input) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/DeviceProxy.cs:line 73
at Microsoft.Azure.Devices.Edge.Hub.Core.Device.DeviceMessageHandler.SendMessageAsync(IMessage message, String input) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Core/device/DeviceMessageHandler.cs:line 256
at Microsoft.Azure.Devices.Edge.Hub.Core.Routing.ModuleEndpoint.ModuleMessageProcessor.<>c__DisplayClass5_2.<<ProcessAsync>b__0>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Core/routing/ModuleEndpoint.cs:line 98
2018-12-21 07:28:33.669 +00:00 [WRN] - Error sending messages to module bewaPi1/SampleModule2
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at DotNetty.Buffers.HeapArena.NewChunk(Int32 pageSize, Int32 maxOrder, Int32 pageShifts, Int32 chunkSize)
at DotNetty.Buffers.PoolArena`1.AllocateNormal(PooledByteBuffer`1 buf, Int32 reqCapacity, Int32 normCapacity)
at DotNetty.Buffers.PoolArena`1.Allocate(PoolThreadCache`1 cache, PooledByteBuffer`1 buf, Int32 reqCapacity)
at DotNetty.Buffers.PooledByteBufferAllocator.NewHeapBuffer(Int32 initialCapacity, Int32 maxCapacity)
at DotNetty.Buffers.AbstractByteBufferAllocator.Buffer(Int32 initialCapacity, Int32 maxCapacity)
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.ByteBufferConverter.ToByteBuffer(Byte[] bytes) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/ByteBufferConverter.cs:line 39
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.ProtocolGatewayMessageConverter.FromMessage(IMessage message) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/ProtocolGatewayMessageConverter.cs:line 101
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.DeviceProxy.SendMessageAsync(IMessage message, String input) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/DeviceProxy.cs:line 73
at Microsoft.Azure.Devices.Edge.Hub.Core.Device.DeviceMessageHandler.SendMessageAsync(IMessage message, String input) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Core/device/DeviceMessageHandler.cs:line 256
at Microsoft.Azure.Devices.Edge.Hub.Core.Routing.ModuleEndpoint.ModuleMessageProcessor.<>c__DisplayClass5_2.<<ProcessAsync>b__0>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Core/routing/ModuleEndpoint.cs:line 98
2018-12-21 07:28:39.012 +00:00 [WRN] - Non retryable exception occurred while sending message.
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at DotNetty.Buffers.HeapArena.NewChunk(Int32 pageSize, Int32 maxOrder, Int32 pageShifts, Int32 chunkSize)
at DotNetty.Buffers.PoolArena`1.AllocateNormal(PooledByteBuffer`1 buf, Int32 reqCapacity, Int32 normCapacity)
at DotNetty.Buffers.PoolArena`1.Allocate(PoolThreadCache`1 cache, PooledByteBuffer`1 buf, Int32 reqCapacity)
at DotNetty.Buffers.PooledByteBufferAllocator.NewHeapBuffer(Int32 initialCapacity, Int32 maxCapacity)
at DotNetty.Buffers.AbstractByteBufferAllocator.Buffer(Int32 initialCapacity, Int32 maxCapacity)
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.ByteBufferConverter.ToByteBuffer(Byte[] bytes) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/ByteBufferConverter.cs:line 39
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.ProtocolGatewayMessageConverter.FromMessage(IMessage message) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/ProtocolGatewayMessageConverter.cs:line 101
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.DeviceProxy.SendMessageAsync(IMessage message, String input) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/DeviceProxy.cs:line 73
at Microsoft.Azure.Devices.Edge.Hub.Core.Device.DeviceMessageHandler.SendMessageAsync(IMessage message, String input) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Core/device/DeviceMessageHandler.cs:line 256
at Microsoft.Azure.Devices.Edge.Hub.Core.Routing.ModuleEndpoint.ModuleMessageProcessor.<>c__DisplayClass5_2.<<ProcessAsync>b__0>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Core/routing/ModuleEndpoint.cs:line 98
2018-12-21 07:28:39.015 +00:00 [WRN] - Error sending messages to module bewaPi1/SampleModule2
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at DotNetty.Buffers.HeapArena.NewChunk(Int32 pageSize, Int32 maxOrder, Int32 pageShifts, Int32 chunkSize)
at DotNetty.Buffers.PoolArena`1.AllocateNormal(PooledByteBuffer`1 buf, Int32 reqCapacity, Int32 normCapacity)
at DotNetty.Buffers.PoolArena`1.Allocate(PoolThreadCache`1 cache, PooledByteBuffer`1 buf, Int32 reqCapacity)
at DotNetty.Buffers.PooledByteBufferAllocator.NewHeapBuffer(Int32 initialCapacity, Int32 maxCapacity)
at DotNetty.Buffers.AbstractByteBufferAllocator.Buffer(Int32 initialCapacity, Int32 maxCapacity)
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.ByteBufferConverter.ToByteBuffer(Byte[] bytes) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/ByteBufferConverter.cs:line 39
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.ProtocolGatewayMessageConverter.FromMessage(IMessage message) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/ProtocolGatewayMessageConverter.cs:line 101
at Microsoft.Azure.Devices.Edge.Hub.Mqtt.DeviceProxy.SendMessageAsync(IMessage message, String input) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Mqtt/DeviceProxy.cs:line 73
at Microsoft.Azure.Devices.Edge.Hub.Core.Device.DeviceMessageHandler.SendMessageAsync(IMessage message, String input) in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Core/device/DeviceMessageHandler.cs:line 256
at Microsoft.Azure.Devices.Edge.Hub.Core.Routing.ModuleEndpoint.ModuleMessageProcessor.<>c__DisplayClass5_2.<<ProcessAsync>b__0>d.MoveNext() in /home/vsts/work/1/s/edge-hub/src/Microsoft.Azure.Devices.Edge.Hub.Core/routing/ModuleEndpoint.cs:line 98
So the problem is caused by System.OutOfMemoryExceptions. But what is causing those exceptions?
Update2: This seems to be a known problem (see here) and can be solved by Setting OptimizeForPerformance to false. I will try this and see if it works.
Solution
Seems the issue related to https://docs.microsoft.com/en-us/azure/iot-edge/troubleshoot#stability-issues-on-resource-constrained-devices
Please refer the link and turn off OptimizeForPerformance in deployment manifest.
Answered By - Wei
