My first steps with Shinobi (and some questions, sorry).

  • Hello,

    I just signed up here, after having two pretty exhausting days with Shinobi. Where do I start…

    I've been using Zoneminder forever. that is, from the early days somewhere back in 2003 or 2004, don't remember exactly.
    It is, in my opinión, an awkward software package.
    Back in the days there was nothing else that even came close to a full fledge CCTV solution, so I had to put up with all the annoyances.

    The last year I've been actively looking to replace Zoneminder for something else. Blue Iris was a candidate, but it runs on Windows only.
    I already have my Ubuntu server running some stuff, and it has been rocksolid for years and served me well.
    Also, because of the electricity cost in my country, I would not like having to build yet another server (Windows) only for Blue Iris.
    The Linux based CCTV solutions are either lacking or have absurd licensing policies.

    So I have been lurking around the web and saw Moe actively "promoting" Shinobi and he seems pretty convinced it is going to be the next big thing.
    I followed the conversations on ipcamtalk and was shocked to see all the Blue Iris groupies getting upset while Moe explained the pros of Shinobi.

    I'm running Ubuntu 16.04 (17 is not LTS and my Unifi software is not yet compatible with 18.04) so for now it will have to do.
    The plan was initially to test Shinobi side by side with Zoneminder.
    But the Shinobi install screwed up my existing SQL config and Zoneminder was no more (was that on purpose, Moe?)
    So I had to make Shinobi work in one way or the other.
    The next problem was that the old SQL install provoked errors in Shinobi. I was unable to login as the SQL part was totally borked.
    Finally I had to totally purge SQL from the system and start from scratch.
    This time around it worked, but I still had to apply the native password fix that Moe posted on Reddit somewhere.
    So if you have some kind of SQL database on your box, backup and purge. There's no other way.

    After that I had to figure out the delegated admin and sub account structure, for the life of me I just couldn't grasp how to create a sub-account.
    Eventually I got it figured out.

    After to days of full scale war with Shinobi I came to think that it was even more awkward than Zoneminder.
    Documentation is seriously lacking and there is bits and pieces of info all over the web.
    Shinobi doesn't have a large userbase yet and there is no online community (don't use Discord btw), I think that's my main concern.

    But that may change in the future, so I'm willing to stick with Shinobi as I really believe that it is going to be much better than Zoneminder ever was.

    Now as for the questions. I've solved most of the issues and documented the pitfalls. But two issues remain.
    First, I have a mix of cameras and some are MJPEG EasyN. When recording to MP4 with libx264, the videos play just fine with potplayer on Windows, but the power viewer does not.
    The video grid just shows black thumbnails.
    Life view works fine though, it shows in the dashboard.
    I have another cam that is H264 FHD (set to "copy"), and it plays just fine in power viewer and shows in the video grid.
    I have compared the files and the only difference I see is the encoding profile. The quality setting for the MJPEG cam is "1" (maximum) and viewing the file info, the encoding
    profile is "high".
    Can that be the cause that Shinobi won't play back the videos from the MJPEG cam? I would not like to lower the quality.

    As a temporary fix, I've set it to record to WebM with libvpx, also with quality "1" and this works fine...!

    The second question is about browser compatibility.
    On Chrome based browsers like Opera, Vivaldi, Chrome above scenario works fine.
    But if I try the video grid or powe viewer on Edge or IE, I get "Invalid source" and the video thumbnails are all black.
    On forementioned browsers neither cam will play back the videos, regardless of the format used.

    On a final note, the translations need work, the Dutch one made by Yandex is horrible so that will have to be corrected by hand.
    When I get the time, Spanish is next.


  • As a follow up:

    One day with Shinobi and it seems running stable in the sense that there are no lockups, no loss of video and no glitches (that I didn't cause myself).
    I found out that the storage option in the config.json is not being respected.
    I've got a 4TB HD reserved for camera footage, so I modified the config.json to have the second storage option pointing to the 4TB HD.
    But Shinobi just happily keeps recording to the SSD where the OS has been installed. Bummer.

    I solved that by making a symbolic link to the 4TB HD and now the videos are being saved there.
    Maybe a bit strange to see the permissions on the videos set as 777, I would like it a bit more restrictive.

    The VP8 codec + WebM container is a big success so far and an enormous advantage over Zoneminder.
    Even the latest version of ZM with H.264 encoding made 60MB files for 10 minute videos.
    Now, with VP8 and max quality, the files are down to 11MB for 15 minutes with far better quality as compared to ZM.
    Note that the cam is low res, MJPEG 640x480.

    As far as the CPU usage, there are only two cams connected to the server at the moment. One FHD and the VGA MJPEG.
    On the same server, I've got a webserver, plex media server, unifi controller and a couple of samba shares.
    Total CPU usage is now at 6%.
    It's an i3 4th gen, 16MB of ram and 10TB storage.

    So apart from the wonky start, it's looking good down the road because Shinobi leaves me with more headroom to add cameras and the storage is being used way more efficiently as compared to ZM.

    I was trying to get an Intel-hybrid driver running to enable some kind of hardware acceleration for encoding VP8, as I saw the driver is compatible with Haswell processors.
    Sadly, I could not get it to install.
    As I'm running an LTS Ubuntu, there is no hope that these kind of packages become available (LTS focuses on stability, not newer versions).

    Thanks Moe for doing an excellent job, hope you can continue making Shinobi better.


  • Another follow-up...

    Having played with it for some days, I can now start documenting glitches, missing/confusing docs, and maybe even bugs. So here it goes.

    • Translations: If you translate a language file when Shinobi is running, it will mix up the original file and translation.
      My dashboard became half Dutch and half English, until Shinobi was restarted - at that point it was normal again.

    • Streaming method for H.264 cams (RTSP), the docs are confusing on what to use actually.
      The official guide states "HLS", Moe stated in a forum post "VLC gives higher frame rates" and a Facebook post stated "Poseidon" with very few info available.
      Tried VLC on my H.264 cams (I now added another so I have two) but there was a black picture only. VLC does not work at all (it did work on ZM on the same server, and VLC also displays the streams just fine).
      Tried Poseidon, it works for the desktop but on Android there is only a black square.
      Also, on my second H.264 cam I get framerate drops to less than 1fps with this enabled.
      Tried Base64 and it worked on both desktop and Android, but here it gets interesting...As soon as I add the second H.264 cam and choose base64 as streaming method, my CPU usage doubles to 12%. That does not happen with only one camera.
      Finally came back to HLS and it works on both desktop and mobile, but there is a steep delay and the delay is different for each cam, making viewing events confusing.
      Would like to learn more about these issues and what is being considered the ideal, most compatible streaming method (don't need audio btw).
      Mind that the MJPEG camera is not affected by above issues.

    • Loss of streaming, but recording is fine. Maybe because of changing streaming methods, the dashboard suddenly displays black squares only or frozen video. Often it can be resolved by reloading the page, but sometimes I have to close the monitor on the dashboard and open it again from the sidebar to have a picture.
      Again, only the H.264 monitors are affected, not the old MJPEG cam.
      Both of the H.264 cameras are dirt cheap Chinese "XiongMai" (XM) cameras, FHD 1080p.

    • Mobile use in general: Shinobi on Chrome for Android works, sort of. But it is a bit awkward. Only one monitor at the time and the web interface is so messed up that monitors open behind the menu and you have to tap at the border to make the monitor visible.
      Also, when fullscreen it is not possible to zoom in on objects, you have to leave fullscreen but then there is greatly reduced space due to the address bar and stuff.
      For me, Shinobi on an Android phone is practically unusable.
      Likewise, the process to make a link on the home screen for Shinobi as documented by the author is probably not meant for Android 5.0 as I can't find the options described.

    • Updating Shinobi: The docs state that we can use "git show" from inside the Shinobi dir.
      When doing "git show" it comes back with "not installed". When installing on Ubuntu Xenial, it comes back with "not a .git directory".
      So actually the suggested command is not working at all, and it's hard to imagine that git is distro specific.

    • Video storage location not respected: As reported earlier, I can put anything I like in the config.json, it is not possible to relocate video storage to another drive. Shinobi just happily makes a new folder in the same Shinobi dir ("videos2") and records to that folder.

    • Forum hasn't got an edit function. Sometimes I'd like to have the change to correct, rephrase or add something. This is currently not possible.

    That's it for now, it would be really nice if others gave their input also, remember that others may benefit from stuf that we document here.
    That said, I would like to volunteer to (re)write some docs. Form what I've seen, there is some outdated/incorrect info.


  • Ok another follow-up because I got some further...

    The issue with FLV not working has been solved. Actually, newer Chrome versions (and thus Vivaldi, what I'm using) block Flash by default.
    I didn't take that into account. When explicitly enabling flash (registry policy edit) it worked straight away and without the high CPU usage.

    But...Flash is on it's way out, and I don't want to depend on plugins that will eventually disappear. I want to be able to open the browser and Shinobi just has to show me my cams, no matter what I'm viewing on.
    So that took me again to "poseidon" stream method.

    This time I got consistent results.
    Mobile now worked also with poseidon, albeit with the same messed up interface.
    The trick was to enable poseidon over websocket. I still could not get it to work by setting it to "http"
    I also made adjustments to the camera stream as suggested in the docs, by changing "main" profile to "baseline" profile.
    It made a huge difference, the stream showed up on my phone. It's also important to note that the wifi connection has to be consistent, in my location, having internet is a luxury.

    So as for the question of "what stream type to use", I found the answer by Moe somewhere searching the docs:

    "We will use a Poseidon as the example because it works the best and has the greatest performance and compatibility out of all the stream types provided by Shinobi."

    To wrap it up, that's my progress for today with three cams recording and the server sits at 6% cpu usage with several other things running.


  • To finish for today, it's 5am - Shinobi and I had a great time all night but now I want to get some sleep.

    I've found the cause of the videos on the dashboard stuttering or freezing, something I reported in a previous post.
    What I did was trying to edit json.conf while Shinobi was running.
    Then the freezing happened.
    It is not enough to simply restart Shinobi.
    What I did to solve this was: Shutdown WinSCP and or Putty completely (I'm using it to edit config files) and only then restart Shinobi.
    Confirmed working, no problems.

    So my conclusion is that Shinobi is extremely picky when it comes to editing files when it runs. Better to edit when it is not running.

    For now the only issue still open for me is the cpu usage of the mjpeg streamer.
    Both base64 and mjpeg stream method have rather high cpu usage. I tested shutting down the mjpeg monitor temporarily and cpu usage fell almost to 2%.
    So even with two 1080p IP cams streaming there is nearly no system load, the aprox 5% cpu load is caused by the mjpeg stream.
    Another reason to dump those mjpeg cams and switch to h.264 cameras, unless Moe comes up with something cool.


  • Scrap my previous post, after some hours of well deserved sleep, I woke up to find the Poseidon stream method stuttering, freezing and skipping.
    Tested both on Android and desktop pc.

    At this point I'm convinced the Poseidon method is too buggy.
    I've changed to base64 over websocket and that's stable - because I had tested it before.
    The trade off, as found in a post on reddit is the high cpu use.
    It's kinda surprising that, after fiddling with it on my own, I come to the exact same conclusion as this poster:

    "On the flakiness of the web GUI: as per this page's suggestion, I tried JPEG as the stream type. This seems to be more reliable, but, as it says, this enables the JPEG API, so now I have three ffmpeg processes at >=30% CPU each. That's no good. Base64 over Websocket also kills the CPU and the monitor only shows about 1/4 of the camera's frame. MJPEG looks good but kills CPU. So far FLV seems pretty good, but again, it's not documented."

    I came to exactly the same conclusion.
    As said already, FLV for me is no-go because it will soon be blocked al together, at least on Chrome.
    So for now, as it stands, I'll stick with what works but that means my Shinobi box sits at 20% cpu with only 3 cams.

    Frankly, without an efficient streaming method that doesn't kill the cpu, how many cameras can I add before killing the box? For now it seems that 14 cameras is the maximum amount on a core i3 4th generation, and that's actually disappointing.
    Shinobi needs to be polished way more as it is not yet ready for prime time, I hope Moe reads this and can continue improving, as I really like the concept.


    Edit: So I found the edit button. Sorry I'm a bit slow to grasp stuff sometimes but you noticed already.
    Finally I settled for MJPEG streaming out. It's compatible with my Android phone and no skips or freezes.
    The cpu usage is a tad lower, I'm now on 15% cpu for three cameras. Still to high for my liking but it will do for now.

    I plan to upgrade to Ubuntu 18.04 LTS this weekend, as there is a fix for the Unifi software on 18.04.
    I'm silently hoping this will improve Shinobi's performance. Has anyone seen any difference between 16.04 and 18.04 performance wise?

  • I'm back 🙂

    Today I upgraded the Ubuntu server box from 16.04 to 18.04.1.
    It took a while to get everything up and running again, but eventually all is working including Shinoobi.

    While upgrading, also took node.js to version 10, MariaDB version 10.3 and a new ffmpeg (non static) version.
    I could retain the database so no data was lost.

    It's now time for the bug reports, as I've played extensively with Shinobi and believe there are some issues.
    First, the obvious issue is the static ffmpeg that's being checked for and downloaded.
    This "static" ffmpeg does not support hardware acceleration. In my opinion, if there is no vaapi support the ffmpeg should not be used with Shinobi.

    Second, I've compiled from source the Intel hybrid driver to make hardware acceleration work (it's stated however that 18.04 does that out of the box).
    Not that it matters anything, even AFTER changing ffmpeg (and symlinking it) for vaapi support, I came to the conclusion that Shinobi cannot use HW acceleration, at least NO vaapi and NO QS.
    The logs are spitting all kinds of errors about missing syntax, non-specified hw devices etc...
    That's really a shame, because it's very efficient on Intel processors.

    Mind you, I did not test with nvidia cards as my box only uses the on-die Intel graphics.

    Above is perhaps not such a big deal when we do direct-to-disc recording with h.264 cams, but I've got some old mjpeg cams that encode to vpx (or h.264 for that matter) and I'd love to see HW acceleration working on that.

    Third, the mjpeg streaming proved to be stable, albeit with high cpu use. I'd like to switch to poseidon or HLS, but the first one skips and stutters after some time and the second produces a steep delay when live viewing.
    The mjpeg streaming is not free from issues however, when you double click on a monitor to view it full screen and then press escape to return to the dashboard, all the cams go black instantly.
    It takes several minutes for the cams to come up again. Sloppy.

    For now I feel like having tested a lot of combinations and versions and even over two different Ubuntu versions, but I can't get cpu usage any further down. Next stop are the bug reports on this board where I'll list the various issues found.

    So that kinda finishes my testing over the last few days, my conclusion is that Shinobi is amazing, modern and good looking (on desktop) but it's way to buggy still.
    Nevertheless, I'll keep it on my box hoping that it will get better over time. No more awkward zoneminder.


  • Not finished, sorry. Just wanted to share my latest findings from yesterdays upgrade until today.

    First, I came to the conclusion that if the camera is mjpeg you can only use "mjpeg" or "base64 over websocket" as streaming mode.
    No other method will work, so that's just to save people some fiddling with it.
    It's also in the docs, take it literally.

    Next, a word about dirt cheap Chinese H.264 cams.
    As some of you may know, the Chinese design some camera concept and then a lot of Chinese manufacturers can build those cams under license.
    My cameras (or more correctly "camera modules") are from XiongMai brand, this may not mean a lot right now but it is a pretty big company.
    The modules are being manufactured in great numbers and can be integrated in various camera designs.
    Most of them are HiSilicon based, cheap but acceptable quality.

    Where I live we have limited access to technology. It's a third world country, and buying the neat stuff like Axis or Indigo Vision is out of the question.
    Chinese cameras have flooded the market, are relatively cheap and readily available.
    That's how I got mines, for about 50 dollars a piece, FHD.

    The cameras work fine with Shinobi but there are some pitfalls:
    The so called "H.264+" or "H.265+" option will lead you to believe that it offers better compression over the regular H.264/H.265 codecs.
    From my testing and when it comes to Shinobi, that simply does not apply.
    Those "+" extensions do not offer any benefit at all! No smaller files, if that's what you thought.

    The so called "+" extensions do work, but they work with DVR's that are made by the same technology provider.
    The "+" extensions are also called "smart codecs" and provide reduced storage space when enabled.
    This works for example when you are recording static images that don't change much, like an empty office.
    One hour of video normally takes up between 800MB and 1GB, with the "smart" option enabled it gets reduced to only 60MB for H.264+ and 30MB for H.265+.

    So granted it works on DVR's that have support for it, but from my testing with Shinobi there wasn't any improvement, so I figured I could as well further test without the "smart" option enabled.
    Likewise, I disabled motion detection on the camera as it gave very weird results like frames skipping into the past.
    As a last step, I followed the doc on optimizing the RTSP camera and reduced the quality.

    Then I re-enabled the poseidon streaming method on Shinobi. And guess what...
    Next morning still totally stable operation. Actually, I discovered why I had issues with poseidon.
    Poseidon seems to be some kind of play list, segments that are "stitched" together.
    When the quality of the camera is set to high, you'll get skipping, freezing and stuttering.
    With the help of the rtsp camera optimization, I could get it to do reasonable fluent streaming.

    I say reasonable, because on full screen you can still barely see the reloading circle every one or two seconds.
    So it's more or less a trade off, you can use poseidon but don't go to high on the video quality because it will bug out.
    Btw, if I set my cams as suggested in the optimization doc, my cams look like shit.
    There is simply no way I can lower it to 1Mbit/s and keep it neat.
    But that is not Shinobi's fault, it is simply the camera module's quality. But what do you expect for 50 dollar?
    You get what you pay for.

    That said, I believe that if you want to get the most out of Shinobi, you'll need to investigate a bit on what cameras you are going to use.
    The cameras that will suit you are those that can do good quality on low bitrates. And as a lesson, don't look at the mp numbers, the cameras have several components that are important, image sensor, chipset and lens!

    I hope that this clears up some doubts most newbies will have, I'm glad to be able to document my adventures on the bitter low end of the budget.


  • Ok, it's been over a week now with Shinobi, wanted to give a small update with some issues still present.

    First, I'm rather disappointed to see that Moe has not been online since August.
    Why create a Shinobi forum only to let it go abandoned? That's a shame, moreover because he commented (rightfully so) on the lousy Zoneminder forum.

    So how does it run over a week? Well, pretty stable. Only two issues left at this point.
    The poseidon streaming method, though using little cpu, has it quirks.
    When opening the browser and the Shinobi dashboard, the cams with poseidon streaming have stuttering video.
    The only way to resolve this is to close the monitors and reopen them again from the sidebar.
    Likewise, when the monitors are displaying a black screen, it is necessary to reload the page.
    Although not a deal breaker, it's not polished either and it should not happen.
    It's really a bit of a "beta" feeling.

    Second issue I haven't been able to tackle is the automatic deletion of videos after xx days.
    I've set it at 10 days, and the events in the database are being deleted correctly, but the actual videos remain on disk.

    Now this may have to do with me symlinking the storage folder.
    Previously I commented that the second storage option in the "config.json" was not respected.
    I set it to record to another (4TB) disk, but Shinobi happily kept recording to the internal SSD drive.
    So in an effort to force it to record to the 4TB disk, I made a symlink to the drive and it worked fine.
    That however could be the reason that cron does not delete the videos, maybe it doesn't follow symlinks?

    I'm too much of a n00b to grasp what is happening here so I'll just delete the vids manually until Moe one day comes around and throws me a bone.

    Other than the above issues, Shinobi has been working great.


  • I'm baaaaack!

    So nearly 600 views and nobody even bothered to post their findings? Weird community we have here.
    Moe has not been around either.

    A little update after a few months with Shinobi. Definitely not going back to Zoneminder, that's for sure.
    They can keep their awkward software.

    After a few updates I've got Shinobi working stable at last.
    The streaming now works without stuttering.
    The auto delete is also working, albeit not the "delete videos older than xx days" settings. That has never worked.
    But on the positive side, I did set a specific storage amount and Shinobi does respect that, it never goes over 89% and deletes oldest videos automatically.

    Now it's just a matter of getting more cameras to see how Shinobi scales. I don't expect issues.
    I would like to see a better interface for the mobile phone. From what I read, there is a "v3" interface in the making, so lets see what that brings.
    For now, I'm a happy user and hope that Moe can keep working on the project.



Looks like your connection to Shinobi Forum was lost, please wait while we try to reconnect.