Poptartica 12 Posted March 21, 2015 Share Posted March 21, 2015 Hey everyone, hope all is well! So far my MB server has been doing great. This time it's on a Linux Mint 17.1 install. The version I have now (3.0.5557.0) had a lot of improvements that have really made a difference, awesome job devs!! I noticed some other posters were having their Ubuntu servers stopping unexpectedly and/or crashing. I have noticed this occasionally with my Linux Mint's install as well, which may possibly be getting a similar issue. I didn't see logs on some of the posts so hopefully this will be helpful if anyone else is getting something similar: It looks like my server is shutting down with Out of Memory errors (in addition to some other fodder which doesn't quite crash the server but may well be of note). I've attached a server log which contains them and anything else that may be relevant (and, of course, if you notice any other errors/issues that I can correct that would be welcome as well!). I should note that I have my own Mono install which I think is the latest (3.12 last I checked?), could this be the issue? I should also note that I was having issues with excessive memory consumption and eventual server choking on my Ubuntu 14.04 install as well (with the provided Mono package/version). server-63562454889.txt Link to comment Share on other sites More sharing options...
thefirstofthe300 291 Posted March 21, 2015 Share Posted March 21, 2015 I should note hat I have my own Mono install which I think is the latest (3.12 last I checked?), could this be the issue? I should also note that I was having issues with excessive memory consumption and eventual server choking on my Ubuntu 14.04 install as well (with the provided Mono package/version). Yeah, the first thing that I would try is downgrading Mono. If the problem persists, let us know. Link to comment Share on other sites More sharing options...
Poptartica 12 Posted March 23, 2015 Author Share Posted March 23, 2015 (edited) Hey guys - just wanted to follow up on this in case anyone else was having a similar problem. I downgraded my Mono (was 3.12) and that fixed the Out of Memory / server just crashing problems for me. It has stayed up multiple days at a time for me (until the server itself has been restarted of course) with no problems on that front, and is able to return to the proper memory consumption as well. I've been getting some other errors but these errors have been something I've encountered across all versions - System write errors - and they don't crash the server, mostly just make video playback problematic until the user tries again or etc.. I'll come back later with some logs if the issue persists edit: well, it looks like going to mono 3.10 is causing me to get the dropped subtitles problem again Edited March 23, 2015 by Poptartica Link to comment Share on other sites More sharing options...
GrahamH68 3 Posted April 7, 2015 Share Posted April 7, 2015 (edited) Like others, I'm having stability problems with Emby on Linux and have been ever since I started using it on Linux a year or so ago. It's a shame that we can't have a Linux native build instead of having to rely on Mono, which afaik seems to be the root of the problems. What would be involved in that? How can we help? (I'm no expert coder btw). Right now this is the only thing that's stopping me from ditching Plex, which as others have noted "just works", but I so much prefer the functionality, interface, and "ethics" of Emby over Plex. Edited April 7, 2015 by GrahamH68 Link to comment Share on other sites More sharing options...
Luke 37065 Posted April 7, 2015 Share Posted April 7, 2015 Microsoft is putting their weight into mono so it's going to keep getting better and faster Link to comment Share on other sites More sharing options...
gsnerf 26 Posted April 7, 2015 Share Posted April 7, 2015 It's a shame that we can't have a Linux native build instead of having to rely on Mono, which afaik seems to be the root of the problems. What would be involved in that? How can we help? (I'm no expert coder btw). What you see is the typical range of multi-platform problems that happen to any project that do not have a long working multiplatform environment (native or not doesn't matter). Seeing as emby is growing fast feature wise and we have a rather small and new linux developer and packager base these kind of problems are to be expected until most of the points where the platforms behave differently have been ironed out. I'm actually very glad this project is based on a language that is mostly multi-platform to begin with, as there would be growing gaps featurewise between the platforms. If you want to help you are welcome to thourougly test every aspect of emby on linux and pinpoint the source of each problem so it can be fixed 1 Link to comment Share on other sites More sharing options...
thefirstofthe300 291 Posted April 7, 2015 Share Posted April 7, 2015 I agree with gsnerf. If the dev team had decided to use something like C++, the feature gap would be growing. Not too mention development wouldn't be happening at the rate that it is. If you look at Plex, the reason that they chose Java for their development base I am sure had to do with the fact that practically every device on the planet supports Java. C# is a language that is already extremely stable on Windows as it has been around for years. Since Microsoft open-sourced the code, it is now possible to port an extremely stable and fairly well written language to platforms such as Linux, OSX, and even BSD. Since Linux and BSD are pretty much the base for every NAS system, it is totally possible to be like Plex with support for practically every device in existence. Personally, I think the .NET/Mono base has the potential to be better than Java since I trust Microsoft to build better software more than I do Oracle...(However, I am not too fond of Windows as an operating system. ) With Mono 4 on its way, I think we will start to see massive improvements with the Mono engine although it is probably going to be rough for a few releases due to the massive changes to implement the Microsoft source. Now if only I could figure out what is causing some random issues with Debian/Ubuntu. :/ 1 Link to comment Share on other sites More sharing options...
Poptartica 12 Posted April 9, 2015 Author Share Posted April 9, 2015 Oh! looks like you guys have been thinking about this situation while I was away Just thought I'd pop in and put a little update that I've gone back to using Mono 3.12 because I couldn't for the life of me get the subtitles to work right on 3.10. It's actually pretty stable for me now, on rare occasion (maybe once or twice in a very long time) I've restarted the MediaBrowserServer to kill some errant processes. However, for the most part, it's actually worked fairly well for me so far. I ended up clearing out my Mediabrowser package and ANY reference to Mono whatsoever (which included Sonarr). I installed Mono 3.12 before anything else, then Mediabrowser (without the mono-related repositories of course, in case anyone else wanted to do the same), then Sonarr, and everything seemed to work much better as opposed to than grabbing 3.10 and then trying to go to 3.12 after. Link to comment Share on other sites More sharing options...
Jackrats 4 Posted May 4, 2015 Share Posted May 4, 2015 I've been getting these Out of memory failures as well. I used to run the VM under 4G and never had issues. I'm not sure exactly when the issue started, sometime in January, probably somewhere between 3.0.5518.4-Beta and 3.0.5588.1-Stable. After restarting the server, I'll be able to watch maybe 1 or 2 shows, then any subsequent attempts via either Roky or Web UI fail with the ffmpeg "out of memory" error in the Emby logs. If I run the exact quoted ffmpeg command from the logs, it runs absolutely fine. Just for kicks, I bumped up the VM to 10GB RAM and still see the same issue. System reports that 8GB is available. I think it's a fair bet that the "Out of memory" error being logged is inaccurate. If it were true, I would have expected Linux's OOM killer to kick in at some point and that has never occurred. I'd guess that Curious about lack of updates recently, I noticed the new Emby repo and plugged into that. Of course, it moved stuff around, so I copied my /var/opt data to the new location and chown'ed it to the new user. Emby starts up fine now (on 3.0.5597) but I'm still seeing the pattern of ffprobe reporting "Out of memory" errors in the logs as well. After playing a video for about 10 seconds, stopping it, and then trying to play it again about 2.5 minutes later, the ffmpeg "Out of memory" has returned as well. However I am on Mono 3.10 -- not 3.12. Logs attached, but I see nothing interesting occur between the successful play @ 12:13:52 and the failed play @ 12:16:19. I'm going to enable debug logging and see if anything else interesting occurs. One thing I have noticed though is that Emby seems to be leaking file descriptors -- right now there are 761 pipes open by the process as seen in /proc. server-63566337929.txt Link to comment Share on other sites More sharing options...
Luke 37065 Posted May 4, 2015 Share Posted May 4, 2015 I've been getting these Out of memory failures as well. I used to run the VM under 4G and never had issues. I'm not sure exactly when the issue started, sometime in January, probably somewhere between 3.0.5518.4-Beta and 3.0.5588.1-Stable. After restarting the server, I'll be able to watch maybe 1 or 2 shows, then any subsequent attempts via either Roky or Web UI fail with the ffmpeg "out of memory" error in the Emby logs. If I run the exact quoted ffmpeg command from the logs, it runs absolutely fine. Just for kicks, I bumped up the VM to 10GB RAM and still see the same issue. System reports that 8GB is available. I think it's a fair bet that the "Out of memory" error being logged is inaccurate. If it were true, I would have expected Linux's OOM killer to kick in at some point and that has never occurred. I'd guess that Curious about lack of updates recently, I noticed the new Emby repo and plugged into that. Of course, it moved stuff around, so I copied my /var/opt data to the new location and chown'ed it to the new user. Emby starts up fine now (on 3.0.5597) but I'm still seeing the pattern of ffprobe reporting "Out of memory" errors in the logs as well. After playing a video for about 10 seconds, stopping it, and then trying to play it again about 2.5 minutes later, the ffmpeg "Out of memory" has returned as well. However I am on Mono 3.10 -- not 3.12. Logs attached, but I see nothing interesting occur between the successful play @ 12:13:52 and the failed play @ 12:16:19. I'm going to enable debug logging and see if anything else interesting occurs. One thing I have noticed though is that Emby seems to be leaking file descriptors -- right now there are 761 pipes open by the process as seen in /proc. The server is unable to launch ffmpeg, probably due to permissions. We see this with some systems on occasion but not sure yet why. I think a chmod should fix it. Link to comment Share on other sites More sharing options...
Jackrats 4 Posted May 4, 2015 Share Posted May 4, 2015 Hmmm, yet it was able to start it for the first instance of watching the show? So, I restarted Emby and noticed that it is opening pipe FD's at a rate of about 6 per second! [root@[member="Media"] mblogs]# while true; do ls -l /proc/26412/fd | grep pipe | wc -l; sleep 1; done 414 420 429 435 441 447 453 ^C Out of curiousity, it looks like there is also an ffmpeg in the cache that is old. Can I just delete it out of the cache tree? -rw-rw-r--. 1 Emby Emby-Admins 28714176 May 4 11:36 ./Emby/server/cache/temp/2a759b27-08f6-4b8e-82d4-a09f7dbe1470/ffmpeg -rw-rw-r--. 1 Emby Emby-Admins 23493384 Dec 15 22:38 ./Emby/server/cache/temp/2c39c1b3-2729-465c-8f40-ed4fa3b57082/ffmpeg drwxrwxr-x. 3 Emby Emby-Admins 21 Apr 17 19:29 ./Emby/server/ffmpeg -rwxrwxrwx. 1 Emby Emby-Admins 28714176 Apr 17 18:39 ./Emby/server/ffmpeg/20150331/ffmpeg Link to comment Share on other sites More sharing options...
Jackrats 4 Posted May 4, 2015 Share Posted May 4, 2015 Ok, so debug logging enabled seems to shed some light here a bit (I think). After restarting the server, Emby if extracting (chapter?) images with ffmpeg at about the same rate as the pipe FD leak is occurring: 2015-05-04 12:45:05.7939 Debug - MediaEncoder: /var/opt/Emby/server/ffmpeg/20150331/ffmpeg -ss 00:25:00.000 -i file:"/nas/usenet/Media/TV/The Walking Dead/Season 5/The Walking Dead - S05E14 - Spend HDTV-720p.mkv" -threads 1 -v quiet -vframes 1 -vf "scale=600:trunc(600/dar/2)*2,thumbnail=30" -f image2 "-" 2015-05-04 12:45:06.2802 Debug - MediaEncoder: /var/opt/Emby/server/ffmpeg/20150331/ffmpeg -ss 00:30:00.000 -i file:"/nas/usenet/Media/TV/The Walking Dead/Season 5/The Walking Dead - S05E14 - Spend HDTV-720p.mkv" -threads 1 -v quiet -vframes 1 -vf "scale=600:trunc(600/dar/2)*2,thumbnail=30" -f image2 "-" 2015-05-04 12:45:06.7418 Debug - MediaEncoder: /var/opt/Emby/server/ffmpeg/20150331/ffmpeg -ss 00:35:00.000 -i file:"/nas/usenet/Media/TV/The Walking Dead/Season 5/The Walking Dead - S05E14 - Spend HDTV-720p.mkv" -threads 1 -v quiet -vframes 1 -vf "scale=600:trunc(600/dar/2)*2,thumbnail=30" -f image2 "-" 2015-05-04 12:45:07.1683 Debug - MediaEncoder: /var/opt/Emby/server/ffmpeg/20150331/ffmpeg -ss 00:40:00.000 -i file:"/nas/usenet/Media/TV/The Walking Dead/Season 5/The Walking Dead - S05E14 - Spend HDTV-720p.mkv" -threads 1 -v quiet -vframes 1 -vf "scale=600:trunc(600/dar/2)*2,thumbnail=30" -f image2 "-" This goes on until finally: 2015-05-04 12:47:12.3288 Debug - MediaEncoder: /var/opt/Emby/server/ffmpeg/20150331/ffmpeg -ss 00:40:00.000 -i file:"/nas/usenet/Media/TV/American Horror Story/Season 4/American Horror Story - S04E06 - Bullseye HDTV-720p.mkv" -threads 1 -v quiet -vframes 1 -vf "scale=600:trunc(600/dar/2)*2,thumbnail=30" -f image2 "-" 2015-05-04 12:47:12.8650 Debug - MediaEncoder: /var/opt/Emby/server/ffmpeg/20150331/ffmpeg -ss 00:45:00.000 -i file:"/nas/usenet/Media/TV/American Horror Story/Season 4/American Horror Story - S04E06 - Bullseye HDTV-720p.mkv" -threads 1 -v quiet -vframes 1 -vf "scale=600:trunc(600/dar/2)*2,thumbnail=30" -f image2 "-" 2015-05-04 12:47:12.8650 Error - MediaEncoder: I-frame image extraction failed, will attempt standard way. Input: file:"/nas/usenet/Media/TV/American Horror Story/Seas on 4/American Horror Story - S04E06 - Bullseye HDTV-720p.mkv" 2015-05-04 12:47:12.8650 Debug - MediaEncoder: /var/opt/Emby/server/ffmpeg/20150331/ffmpeg -ss 00:45:00.000 -i file:"/nas/usenet/Media/TV/American Horror Story/Season 4/American Horror Story - S04E06 - Bullseye HDTV-720p.mkv" -threads 1 -v quiet -vframes 1 -vf "scale=600:trunc(600/dar/2)*2" -f image2 "-" 2015-05-04 12:47:12.8650 Error - App: Error extraching chapter images for /nas/usenet/Media/TV/American Horror Story/Season 4/American Horror Story - S04E06 - Bullseye HDTV-720p.mkv *** Error Report *** Version: 3.0.5597.1 Command line: /opt/Emby/server/bin/MediaBrowser.Server.Mono.exe -programdata /var/opt/Emby/server Operating system: Unix 3.10.0.229 Processor count: 8 64-Bit OS: True 64-Bit Process: True Program data path: /var/opt/Emby/server Mono: 3.10.0 (tarball Sat Nov 15 03:35:34 UTC 2014) Application Path: /opt/Emby/server/bin/MediaBrowser.Server.Mono.exe ApplicationName='/var/opt/Emby/server/ffmpeg/20150331/ffmpeg', CommandLine='-ss 00:45:00.000 -i file:"/nas/usenet/Media/TV/American Horror Story/Season 4/American Horror Story - S04E06 - Bullseye HDTV-720p.mkv" -threads 1 -v quiet -vframes 1 -vf "scale=600:trunc(600/dar/2)*2" -f image2 "-"', CurrentDirectory='', Native error= Out of memory System.ComponentModel.Win32Exception at System.Diagnostics.Process.Start_noshell (System.Diagnostics.ProcessStartInfo startInfo, System.Diagnostics.Process process) [0x00000] in <filename unknown>:0 at System.Diagnostics.Process.Start_common (System.Diagnostics.ProcessStartInfo startInfo, System.Diagnostics.Process process) [0x00000] in <filename unknown>:0 at System.Diagnostics.Process.Start () [0x00000] in <filename unknown>:0 at (wrapper remoting-invoke-with-check) System.Diagnostics.Process:Start () at MediaBrowser.MediaEncoding.Encoder.MediaEncoder.StartProcess (MediaBrowser.MediaEncoding.Encoder.ProcessWrapper process) [0x00000] in <filename unknown>:0 at MediaBrowser.MediaEncoding.Encoder.MediaEncoder+<ExtractImageInternal>c__async4.MoveNext () [0x00000] in <filename unknown>:0 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x00000] in <filename unknown>:0 at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[System.IO.Stream].GetResult () [0x00000] in <filename unknown>:0 at MediaBrowser.MediaEncoding.Encoder.MediaEncoder+<ExtractImage>c__async3.MoveNext () [0x00000] in <filename unknown>:0 --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x00000] in <filename unknown>:0 at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[System.IO.Stream].GetResult () [0x00000] in <filename unknown>:0 at MediaBrowser.Server.Implementations.MediaEncoder.EncodingManager+<RefreshChapterImages>c__async0.MoveNext () [0x00000] in <filename unknown>:0 It seems as if the pipes for these instances are not being closed properly. Once the process hits ~820 FD's, it starts giving the Out of Memory error for any additional ffprobe or ffmpeg attempts. Full log attached. server-63566337929.txt Link to comment Share on other sites More sharing options...
Doonga 17 Posted May 4, 2015 Share Posted May 4, 2015 I've been seeing this issue also especially with chapter extraction. Once the ffprobe out of memory errors start no videos will play and the server needs to be restarted to correct the issue. This is on a fresh install via docker, also reproduced on a manual install. I can upload logs once it happens again but it will be very similar to the one above. Link to comment Share on other sites More sharing options...
Doonga 17 Posted May 6, 2015 Share Posted May 6, 2015 (edited) Just a note, this is on Centos 7. Mono 3.10. Server Version 3.0.5597.1. I have disabled chapter extraction for now as it kills my ability to watch anything once it runs for about 30 seconds to a minute. The server needs to be restarted to clear up the file descriptors. Edited May 6, 2015 by Doonga Link to comment Share on other sites More sharing options...
Luke 37065 Posted May 6, 2015 Share Posted May 6, 2015 Since you've identified a specific issue related to image extraction, I've made a few changes to the way the process is disposed after extraction. Please grab the attached zip and unzip over your existing installation. Let me know if it makes a difference. Thanks. MBServer.Mono_3.0.5603.36935.zip Link to comment Share on other sites More sharing options...
Doonga 17 Posted May 6, 2015 Share Posted May 6, 2015 (edited) This looks much much better. I had a bit of a crash course on how to modify a docker image too. I got attached version of the server running and scanned my movie library in and then ran a chapter extraction. I was able to see it pulling images and the file descriptors was fluctuating between 9 and 14 for most of the process. I didn't let it run all of the way through mostly because I don't want to wait that long. Here's the log file if you want to check it out. Thanks very much for looking into this! server-63566463369.txt Edited May 6, 2015 by Doonga Link to comment Share on other sites More sharing options...
Luke 37065 Posted May 6, 2015 Share Posted May 6, 2015 Thanks. This got recently introduced because we had to monitor the processes a little closer to make sure that if you shutdown the server we are disposing them. unfortunately the immediate disposal became a casualty, and at least in other operating systems, takes much longer before any problems arise. Link to comment Share on other sites More sharing options...
Luke 37065 Posted May 6, 2015 Share Posted May 6, 2015 To everyone else, you can expect a new server release in the next day or two to resolve this. Link to comment Share on other sites More sharing options...
mediacowboy 438 Posted May 6, 2015 Share Posted May 6, 2015 Luke will the new server just address the Linux server. Here's why I ask. I am seeing this in Windows as well. http://emby.media/community/index.php?/topic/19476-Emby-OUT-OF-MEMORY-ERRORS Link to comment Share on other sites More sharing options...
Jackrats 4 Posted May 6, 2015 Share Posted May 6, 2015 Thanks Luke! Glad you were able to track this one down, it's been a bear. Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now