Jump to content

Instability ever since emby


embyfan

Recommended Posts

embyfan

First of all I've been a lurker for a long time and used mediabrowser for even longer, I actually contributed to the project a couple times already.

A little background of what I am using is Debian Jessie  and Mono 3.12

 

My problem is that ever since emby was released I have had no stability at all... the server keeps on crashing randomly, sometimes it takes a few hours, sometimes it takes a few minutes.

My last stable was just before when sync was added, I think it was 3.0.5518.7 ( may not be the exact version) 

Here is a the last portion of the log (before it crashes)

2015-03-27 17:42:13.8665 Error - App: Error getting response from http://api.fanart.tv/webservice/newmusic/5c6b04c68e904cfed1e6cbc9a9e683d4/1426226335/&client_key=_______________
	*** Error Report ***
	Version: 3.0.5557.20000
	Command line: /opt/mediabrowser/MediaBrowser.Server.Mono.exe -programdata /media/hdds/buffer/mediabrowser/data
	Operating system: Unix 3.16.0.4
	Processor count: 6
	64-Bit OS: True
	64-Bit Process: True
	Program data path: /media/hdds/buffer/mediabrowser/data
	Mono: 3.12.1 (tarball Fri Mar  6 19:12:47 UTC 2015)
	Application Path: /opt/mediabrowser/MediaBrowser.Server.Mono.exe
	The remote server returned an error: (404) Not Found.
	System.Net.WebException
	  at System.Net.HttpWebRequest.CheckFinalStatus (System.Net.WebAsyncResult result) [0x00000] in <filename unknown>:0 
	  at System.Net.HttpWebRequest.SetResponseData (System.Net.WebConnectionData data) [0x00000] in <filename unknown>:0 
	
2015-03-27 17:42:13.8665 Error - App: Error running postscan task
	*** Error Report ***
	Version: 3.0.5557.20000
	Command line: /opt/mediabrowser/MediaBrowser.Server.Mono.exe -programdata /media/hdds/buffer/mediabrowser/data
	Operating system: Unix 3.16.0.4
	Processor count: 6
	64-Bit OS: True
	64-Bit Process: True
	Program data path: /media/hdds/buffer/mediabrowser/data
	Mono: 3.12.1 (tarball Fri Mar  6 19:12:47 UTC 2015)
	Application Path: /opt/mediabrowser/MediaBrowser.Server.Mono.exe
	The remote server returned an error: (404) Not Found.
	MediaBrowser.Model.Net.HttpException
	  at MediaBrowser.Common.Implementations.HttpClientManager.HttpClientManager+<SendAsyncInternal>c__async4.MoveNext () [0x00000] in <filename unknown>:0 
	--- End of stack trace from previous location where exception was thrown ---
	  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x00000] in <filename unknown>:0 
	  at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[MediaBrowser.Common.Net.HttpResponseInfo].GetResult () [0x00000] in <filename unknown>:0 
	  at MediaBrowser.Common.Implementations.HttpClientManager.HttpClientManager+<SendAsync>c__async1.MoveNext () [0x00000] in <filename unknown>:0 
	--- End of stack trace from previous location where exception was thrown ---
	  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x00000] in <filename unknown>:0 
	  at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[MediaBrowser.Common.Net.HttpResponseInfo].GetResult () [0x00000] in <filename unknown>:0 
	  at MediaBrowser.Common.Implementations.HttpClientManager.HttpClientManager+<Get>c__async0.MoveNext () [0x00000] in <filename unknown>:0 
	--- End of stack trace from previous location where exception was thrown ---
	  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x00000] in <filename unknown>:0 
	  at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[System.IO.Stream].GetResult () [0x00000] in <filename unknown>:0 
	  at MediaBrowser.Providers.Music.FanartUpdatesPostScanTask+<GetArtistIdsToUpdate>c__async1.MoveNext () [0x00000] in <filename unknown>:0 
	--- End of stack trace from previous location where exception was thrown ---
	  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x00000] in <filename unknown>:0 
	  at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[System.Collections.Generic.IEnumerable`1[System.String]].GetResult () [0x00000] in <filename unknown>:0 
	  at MediaBrowser.Providers.Music.FanartUpdatesPostScanTask+<Run>c__async0.MoveNext () [0x00000] in <filename unknown>:0 
	--- End of stack trace from previous location where exception was thrown ---
	  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x00000] in <filename unknown>:0 
	  at System.Runtime.CompilerServices.ConfiguredTaskAwaitable+ConfiguredTaskAwaiter.GetResult () [0x00000] in <filename unknown>:0 
	  at MediaBrowser.Server.Implementations.Library.LibraryManager+<RunPostScanTasks>c__async4.MoveNext () [0x00000] in <filename unknown>:0 
	InnerException: System.Net.WebException
	The remote server returned an error: (404) Not Found.
	  at System.Net.HttpWebRequest.CheckFinalStatus (System.Net.WebAsyncResult result) [0x00000] in <filename unknown>:0 
	  at System.Net.HttpWebRequest.SetResponseData (System.Net.WebConnectionData data) [0x00000] in <filename unknown>:0 
	

I've attached a couple other instances that I have saved ...

 

Can anyone guide me into a resolution? else I am moving back to an older release. Thanks much

 

I also get a lot of errors with fanart. (i've removed my real api key, that is not the problem)

server-63563031710.txt

server-63563091961.txt

Link to comment
Share on other sites

thefirstofthe300

I think the problem may not lie with the server itself but with Mono. Mono 3.12 has had some known stability issues so I would recommend rolling back to Mono 3.10 (been rock solid for me). 

Link to comment
Share on other sites

Since the latest server update mine is actually completely unusable. It crashes before I can even view the dashboard. I filed a bug report on github a couple days ago, I haven't seen any activity on it yet though. I also posted in the Ubuntu section, there have been about ten or so reports of the latest server update not workinng on Linux, but I have seen few solutions.

Edited by biddy
Link to comment
Share on other sites

Keep in mind that the latest stable release is 3.0.5518.7. So if you are looking for stability you should stay with that version.

 

Having said that I'm running the latest beta release 3.0.5557 on CentOS 7 with mono 3.10 and it's working fine for me. I'm transcoding video to iOS devices (iOS app), playing music from the webclient for hours, etc., without problems.

 

So my advice is that you try with mono 3.10 instead of 3.12 and see if that helps. Or you can always go with the latest stable release.

Link to comment
Share on other sites

spootdev

Use 3.10 instead of 3.12 or you'll have bad times.  And tbh I had major troubles with Xubuntu 14.10 and redid my server to Centos 6.6 this weekend and all my stability issues went away and it's much faster for whatever reason.

Link to comment
Share on other sites

steini

I was also on ubuntu 14.04 with constant random crashing.. both on mono 3.10 and 3.12

 

Moved the library over to Centos 7 + mono 3.10 yesterday and it has been stable since.

 

Note that I was using version 3.0.5557.0 on both systems

Link to comment
Share on other sites

embyfan

I've tried everything, even the new update from today (3.0.5569.0) and it still crashes.  I've installed mono 3.10, purged my 3.12, purged and installed Emby.... only time it works for a bit is when I sudo start the server... 

 

Any idea when it will be working back again on debian?

 

does this give any clues?

2015-04-01 00:53:45.7262 Info - App: HttpClientManager POST: http://www.mb3admin.com/admin/service/registration/validate
2015-04-01 00:53:45.7311 Error - DtoUtils: ServiceBase<TRequest>::Service Exception
	*** Error Report ***
	Version: 3.0.5569.0
	Command line: /opt/mediabrowser/MediaBrowser.Server.Mono.exe
	Operating system: Unix 3.16.0.4
	Processor count: 6
	64-Bit OS: True
	64-Bit Process: True
	Program data path: /media/hdds/buffer/mediabrowser
	Mono: 3.10.0 (tarball Sat Mar 28 22:05:14 EDT 2015)
	Application Path: /opt/mediabrowser/MediaBrowser.Server.Mono.exe
	Could not find file "/opt/mediabrowser/dashboard-ui/css/images/clients/androidtv-tile.png".
	System.IO.FileNotFoundException
	  at System.IO.FileStream..ctor (System.String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, Boolean anonymous, FileOptions options) [0x00000] in <filename unknown>:0 
	  at System.IO.FileStream..ctor (System.String path, FileMode mode, FileAccess access, FileShare share) [0x00000] in <filename unknown>:0 
	  at (wrapper remoting-invoke-with-check) System.IO.FileStream:.ctor (string,System.IO.FileMode,System.IO.FileAccess,System.IO.FileShare)
	  at MediaBrowser.Common.Implementations.IO.CommonFileSystem.GetFileStream (System.String path, FileMode mode, FileAccess access, FileShare share, Boolean isAsync) [0x00000] in <filename unknown>:0 
	  at MediaBrowser.WebDashboard.Api.PackageCreator.GetRawResourceStream (System.String path) [0x00000] in <filename unknown>:0 
	  at MediaBrowser.WebDashboard.Api.PackageCreator+<GetResource>c__async0.MoveNext () [0x00000] in <filename unknown>:0 
	--- End of stack trace from previous location where exception was thrown ---
	  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x00000] in <filename unknown>:0 
	  at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[TResult].GetResult () [0x00000] in <filename unknown>:0 
	  at MediaBrowser.Server.Implementations.HttpServer.HttpResultFactory+<GetStaticResult>c__async0.MoveNext () [0x00000] in <filename unknown>:0 
	
2015-04-01 00:53:45.7311 Error - DtoUtils: ServiceBase<TRequest>::Service Exception
	*** Error Report ***
	Version: 3.0.5569.0
	Command line: /opt/mediabrowser/MediaBrowser.Server.Mono.exe
	Operating system: Unix 3.16.0.4
	Processor count: 6
	64-Bit OS: True
	64-Bit Process: True
	Program data path: /media/hdds/buffer/mediabrowser
	Mono: 3.10.0 (tarball Sat Mar 28 22:05:14 EDT 2015)
	Application Path: /opt/mediabrowser/MediaBrowser.Server.Mono.exe
	Nullable object must have a value.
	System.InvalidOperationException
	  at System.Nullable`1[System.Boolean].get_Value () [0x00000] in <filename unknown>:0 
	  at MediaBrowser.Common.Implementations.Security.PluginSecurityManager.get_IsMBSupporter () [0x00000] in <filename unknown>:0 
	  at MediaBrowser.Server.Implementations.Connect.ConnectManager+<GetConnectSupporterSummary>c__async13.MoveNext () [0x00000] in <filename unknown>:0 
	--- End of stack trace from previous location where exception was thrown ---
	  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x00000] in <filename unknown>:0 
	  at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[TResult].GetResult () [0x00000] in <filename unknown>:0 
	  at MediaBrowser.Api.ConnectService+<Get>c__async0.MoveNext () [0x00000] in <filename unknown>:0 
	
2015-04-01 00:53:45.7338 Error - HttpServer: Error processing request for /web/css/images/clients/androidtv-tile.png
	*** Error Report ***
	Version: 3.0.5569.0
	Command line: /opt/mediabrowser/MediaBrowser.Server.Mono.exe
	Operating system: Unix 3.16.0.4
	Processor count: 6
	64-Bit OS: True
	64-Bit Process: True
	Program data path: /media/hdds/buffer/mediabrowser
	Mono: 3.10.0 (tarball Sat Mar 28 22:05:14 EDT 2015)
	Application Path: /opt/mediabrowser/MediaBrowser.Server.Mono.exe
	Could not find file "/opt/mediabrowser/dashboard-ui/css/images/clients/androidtv-tile.png".
	ServiceStack.HttpError
	No Stack Trace Available
	
2015-04-01 00:53:45.7345 Error - HttpServer: Error processing request for /Connect/Supporters
	*** Error Report ***
	Version: 3.0.5569.0
	Command line: /opt/mediabrowser/MediaBrowser.Server.Mono.exe
	Operating system: Unix 3.16.0.4
	Processor count: 6
	64-Bit OS: True
	64-Bit Process: True
	Program data path: /media/hdds/buffer/mediabrowser
	Mono: 3.10.0 (tarball Sat Mar 28 22:05:14 EDT 2015)
	Application Path: /opt/mediabrowser/MediaBrowser.Server.Mono.exe
	Nullable object must have a value.
	ServiceStack.HttpError
	No Stack Trace Available
	
2015-04-01 00:53:45.7357 Error - NotFoundHttpHandler: 192.168.11.41:8096 Request not found: /web/css/images/clients/androidtv-tile.png
Link to comment
Share on other sites

embyfan

I've removed all plugins I had installed and noticed this now:

\

Native stacktrace:

        mono() [0x4b5fc6]
        mono() [0x50d7c3]
        mono() [0x423990]
        /lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0) [0x7f41c50d18d0]
        mono() [0x53cb4e]
        [0x40b8387e]

Debug info from gdb:


=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================

Link to comment
Share on other sites

steini

My Ubuntu crashes were all related to libpthread.so 

 

My Centos 7 install has not had a single hiccup.

 

If you have the option to switch to Centos it's really easy to migrate Emby - just copy the data directory (just remember to fix the permissions if needed)

Link to comment
Share on other sites

embyfan

I've narrowed down to where the instability lies (the new updates seemed to have helped) 

 

Just to make sure I started again with a fresh install of Debian Jessie and mono 3.10. Installed all dependencies for emby and upgraded it to the latest (3.0.5572).

 

The problem lies on whenever a scheduled task that deals with pictures and sometimes the media library scan. 

 

When I hit the "refresh people" or sometimes "chapter image extraction"  one of the 2 things will happen:

1) The server will either stall, still be browse-able, but the logs will stop immediately and when I try and play any media, it will hang on a blank (black) screen, the play bar will not function, but all the metadata will show up, including logo, cover.... ffmpeg will run normally and transcode and save into the temp folder, but apparently the movie won't be streamed. (only tested with web browser).

2) The server will crash completely and will need to be restarted.

 

When I hit the Roku "create thumbnails" option 2 (above) will happen.

When I hit "scan library" option 1 will be likely but doesn't happen very often.

All other tasks do not cause it to crash.

 

Also if those tasks run on schedule, without me pushing it to happen the same scenarios will happen. (no logs show any errors or warnings)

 

Can anyone help?

Edited by embyfan
Link to comment
Share on other sites

embyfan

here are the logs... they all stop as soon as a scheduled scan starts... I even had trouble restarting the server because it wants to do it as soon as I start it.

 

The transcode was stopped by me... I was just testing things around.

 

Thanks for the attention.

 

PS

I compiled my own ImageMagick since dedian  has the Quantum 16 package not the Quantum 8 and 

 

./configure --with-quantum-depth=8

and 

./configure --with-quantum-depth=8 --enable-shared=yes

 

with the same results...

 

Could it be that i should use some other configs such as 

--with-magick-plus-plus=yes

--with-modules=yes

 

or does it need any other configs enabled?

http://www.imagemagick.org/script/advanced-unix-installation.php

2015-04-05 06_48_09.txt

2015-04-05 21_09_10.txt

2015-04-05 21_09_32.txt

2015-04-05 21_11_40.txt

Transcode_2015-04-05 06_49.txt

Link to comment
Share on other sites

thefirstofthe300

A vanilla ImageMagick build should work with Emby right out of the box. In fact, Q16 should work just fine. Q8 is preferred because it is a bit faster with no real loss in image quality.

Link to comment
Share on other sites

embyfan

I do understand that Q16 will do Q8 too, and even made a symlink with the debian default package (Q16) prior removing it and compiling my own.

I am trying and figure this out and rule out as many variables as possible

Link to comment
Share on other sites

thefirstofthe300

I have been doing a bit of research on the libpthread.so error from Mono and the entire thing seems to be related to a problem with Mono on Ubuntu itself. I find it funny that I have not heard any complaints from any other Mono distros. This seems to be related to Debian based distros only.

 

A thread on the Sonarr forums reported that downgrading to 3.6 fixed their issue (same one you seem to be having). Can you try that as I doubt the problem exists in the server itself.

Link to comment
Share on other sites

embyfan

I haven't had that particular issue coming up again since the fresh install, but I'll give mono 3.6 a try and keep u posted

Link to comment
Share on other sites

Please unzip the attached file into your server installation and replace the existing version. There's still no evidence yet of a problem within Emby Server but this is just a test. Thanks.

ImageMagickSharp.zip

Link to comment
Share on other sites

Hi Guys,

 

I can confirm this issue with Debian Jessie, Mono 3.10 and ImageMagick Q16 (from Debian repo). My thread was  http://emby.media/community/index.php?/topic/19847-real-time-monitoring-queued-refreshed-nothing-happens/

 

 

 

@@Luke your new ImageMagickSharp.dll seems to have done the trick for me. Thank you!

Edited by xelar
Link to comment
Share on other sites

embyfan

Sorry for the delay, I had a couple days I was busy and could not test it out.

 

@@Luke here it seems that the dll has fixed the issue, I haven't fully tested or given a proper amount of time yet, but I added some large libraries and its running smoothly, at least for the last couple hours.

 

only one more problem, I was advised by @ to downgrade my mono even further (3.6), and now I am getting this:

2015-04-08 00:58:29.7671 Info - HttpServer: attempting to load pfx: /media/hdds/buffer/mediabrowser/ssl/cert_9c31b7884ea5475c8687970fc5996297.pfx
2015-04-08 00:58:29.7761 Error - HttpServer: Exception loading certificate: /media/hdds/buffer/mediabrowser/ssl/cert_9c31b7884ea5475c8687970fc5996297.pfx
	*** Error Report ***
	Version: 3.0.5572.0
	Command line: /opt/mediabrowser/MediaBrowser.Server.Mono.exe -programdata /media/hdds/buffer/mediabrowser
	Operating system: Unix 3.16.0.4
	Processor count: 6
	64-Bit OS: True
	64-Bit Process: True
	Program data path: /media/hdds/buffer/mediabrowser
	Mono: 3.6.0 (tarball Mon Apr  6 04:55:58 EDT 2015)
	Application Path: /opt/mediabrowser/MediaBrowser.Server.Mono.exe
	Input data cannot be coded as a valid certificate.
	System.Security.Cryptography.CryptographicException
	  at Mono.Security.X509.X509Certificate.Parse (System.Byte[] data) [0x00000] in <filename unknown>:0 
	  at Mono.Security.X509.X509Certificate..ctor (System.Byte[] data) [0x00000] in <filename unknown>:0 
	  at System.Security.Cryptography.X509Certificates.X509Certificate2.Import (System.Byte[] rawData, System.String password, X509KeyStorageFlags keyStorageFlags) [0x00000] in <filename unknown>:0 
	  at System.Security.Cryptography.X509Certificates.X509Certificate2.Import (System.String fileName, System.String password, X509KeyStorageFlags keyStorageFlags) [0x00000] in <filename unknown>:0 
	  at System.Security.Cryptography.X509Certificates.X509Certificate2..ctor (System.String fileName) [0x00000] in <filename unknown>:0 
	  at SocketHttpListener.Net.EndPointListener.LoadCertificateAndKey (System.Net.IPAddress addr, Int32 port, System.String certificateLocation) [0x00000] in <filename unknown>:0 
	InnerException: System.Security.Cryptography.CryptographicException
	Input data cannot be coded as a valid certificate.
	  at Mono.Security.X509.X509Certificate.Parse (System.Byte[] data) [0x00000] in <filename unknown>:0 

Would you advise it's safe to upgrade to mono 3.10?

 

Thanks!

Link to comment
Share on other sites

thefirstofthe300

@hurricanhernandez

 

Could something in the build have gotten screwed up? If the new ImageMagickSharp DLL fixed the problem, I think we need to take a loom at the possibility of the build getting messed up.

Link to comment
Share on other sites

thefirstofthe300

@@daniellog Just to make sure your problem isn't related to this, please replace the ImageMagickSharp in the current installation with the one Luke posted above.

Link to comment
Share on other sites

daniellog

@ I have done that and it works with the mediabrowser user until I try to playback a file and it doesn't work. Changing back to root seems to fix it, it could be more an issue related to my system. I have all my media on a zfs mount but the permissions are fine and the medaibrowser user has been added to the group to be able to access the media. 

Link to comment
Share on other sites

thefirstofthe300

Can you post a complete log file from when you attempt to playback a file?

 

Also, what are the permissions on the media files? Also, does the correct user/group own the root directory of you media library (not root). Also, if you mount the zfs system in fstab, what is the fstab?

 

If running as root is able to fix your problem, you have a permissions issue somewhere. The question is where.

 

Also, did you change the transcoding temporary directory under Playback->Transcoding? If you did, does the correct user/group own that directory too?

 

EDIT: Also, did you change the permissions on the new ImageMagickSharp??? You probably did but that could very well be causing a problem if you didn't.

Edited by DaBungalow
Link to comment
Share on other sites

thefirstofthe300

@@embyfan If the new ImageMagickSharp fixed the issue, you will need to upgrade again if you want to use SSL. If not you can ignore the error. Only upgrade to 3.10 though. :)

Link to comment
Share on other sites

embyfan

I'e been running it for a while now, but I still have issues. The dll solved the crashing for the most part, but it still "stalls" when I run more than one scheduled task at once. Even if it starts automatically,

The logs will stop, the scan will fail and I am not able to play media, although it successfully gets trans coded.

 

Can we get some help ?

 

Thanks

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...