XaiJu
trapexit

trapexit

patreon


trapexit posts

mergerfs v2.40.1 released

https://github.com/trapexit/mergerfs/releases/tag/2.40.1

Fixes the EIO error experienced by some exporting mergerfs over NFS. Turns out it was both a kernel bug and a mergerfs bug. Hence why it took so long to track down.

For those who used export-support=false in v2.40.0 please stop doing so.

View Post

mergerfs v2.40.0 released

https://github.com/trapexit/mergerfs/releases/tag/2.40.0

Provides a workaround for those using NFS and getting EIO errors. Set the new option `export-support=false`.

It is caused by an apparent kernel bug. Till the bug is fixed this seems to be the only workaround (besides not using NFS). More details are in the README in the new "Remote Filesystems" section.

View Post

mergerfs v2.39.0 released

https://github.com/trapexit/mergerfs/releases/tag/2.39.0

Main feature is the addition of an experimental LD_PRELOAD library which can be used with dynamically linked software to improve file IO by leveraging mergerfs to create/open the file, and then reopening the original file.

htt...

View Post

mergerfs v2.38.0 released

View Post

mergerfs v2.37.X released

  • readdir policies: func.readdir=X where X is seq, cor, or cosr.seq: sequential - works same as before. Iterate over each branch in order configured.
    cosr: concurrent open sequential read - dispatches requests to open directories to a thread pool and then reads directories sequentially the same way seq does.
    cor: concurrent open and read - dispatches requests to a thread pool which both opens and reads directories concurrently.
    thread pool sizes are configurable. cosr:5 would giv...

    View Post

mergerfs 2.36.0

https://github.com/trapexit/mergerfs/releases/tag/2.36.0

Change summary:

  • Allow user to define depth of process thread message queues. Previously these were unbounded and "FORGET" message floods would bloat memory.
  • O_DIRECT now supported, mostly. If software attempt to change the O_DIRECT flag on the fly using fcntl(F_SETFL) it won't work as that is not supported by FUSE....

    View Post

mergerfs 2.35.1 released

Small patch release fixing an inadvertent change in threading default value. 


https://github.com/trapexit/mergerfs/releases/tag/2.35.1

View Post

mergerfs v2.35.0 released

https://github.com/trapexit/mergerfs/releases/tag/2.35.0

A big list of "quality of life" features.

  • Lots of documentation changes and tweaks.
  • Add ability to log to syslog. Used only for certain new features at startup.
  • New FUSE message processing mode. Allow for a separation between threads which are reading messages and those processing messages. See process-thr...

    View Post

Still here... still working...

There hasn't been much in the way of public updates for mergerfs the past several months. I did start working on major rewrites of mergerfs for v3 but got side tracked with 3DO related projects. Mostly tooling to help with media conversion as well as an easier to use development toolkit. I hope to get back to mergerfs soon.

View Post

mergerfs v3.X discussion thread

https://github.com/trapexit/mergerfs/discussions/874

Have listed out some ideas I have for mergerfs v3.X. Please check it out and throw in some ideas if you have them.

View Post

mergerfs 2.32.2 released and general news

I apologize for posting here later than other places.

I've released 2.32.x recently with a few internal updates and bug fixes.

https://github.com/trapexit/mergerfs/releases

There hasn't been much going on publicly with mergerfs but I have spent a bunch of time playing with different ideas. I'm an iterative style developer where I tend to rewrite features a few times before committing them. ...

View Post

mergerfs 2.31.0 released

https://github.com/trapexit/mergerfs/releases/tag/2.31.0

Took a bit longer to get out than I had hoped. I was fooling with a few other features that I considered including but since those are still baking I figured I'd release it as it was.

Nothing too exciting.

Summary:

  • Lots of cleanup
  • new policies: prfd, eppfrd, msppfrd
  • per branch minfreespace
  • ...

    View Post

mergerfs 2.30.0 released

https://github.com/trapexit/mergerfs/releases/tag/2.30.0

Features

  • inodecalc: new argument that allows controlling how inodes are calculated. Can help with NFS stale file handle errors.
  • nfsopenhack: new argument that helps working around issues with  certain issues with NFS not being fully POSIX compatible when exporting  mergerfs.
  • new msp policies: "...

    View Post

scorch v1.0.0 released + updates

Hope everyone is well. Sorry for lack of updates lately. Been doing a lot of random experimentation but haven't made too much progress with any particular project of mine.

But it wasn't all just messing around. I finally did a proper release of scorch with a few new features.

 https://github.com/trapexit/scorch/releases/tag/1.0.0  

  •     live append...

    View Post

Over 1K Stars on GitHub for mergerfs

Thank you to patrons. Thank you users.

View Post

3DO updates

For those interested in my 3DO related work...

1) Created a new website in collaboration with Bugo The Cat / Optimus to document as much as possible about the 3DO and M2 with regards to development. Both for the system itself as well as for emulating the systems. Optimus is working on a homebrew tutorial and created a tutorial for creating 3DO compatible Cinepak videos.

2) I've been working on adding some feat...

View Post

Self Hosted Podcast discusses mergerfs

View Post

Podcast appearance: Extras Show

This is a few weeks old but I was on the Jupiter Network's Extras Show talking about mergerfs back in October. I go into some of the technical aspects of mergerfs. Might be interesting for some.



View Post

mergerfs mentioned in new paper on ExtFuse

https://www.usenix.org/conference/atc19/presentation/bijlani


Ashish Bijlani and Umakishore Ramachandran of Georgia Institute of Technology have been working on an extension to FUSE to allow using eBPF which can help move some of the high latency logic that often slows down a FUSE based filesystem into the kernel and thereby improving performance.

I'm still digging into t...

View Post

2019-06-06: Future plans w/ mergerfs

I've added a bunch of new features and enhancements lately and I've got a list of things to investigate. Wanted to lay them out here and see if there are anything users might be looking for.

* readdir_plus support: Will be easy to add. Probably come in 2.29. Should improve performance of `ls -lh` and the like. It combines the request for files with the metadata rather than separate steps.

* increasing readdir message size: The FUSE kernel module currently hardcodes the message siz...

View Post

mergerfs 2.28.0 released

 

Forgot to announce 2.26 and 2.27 on Patreon. Better late than never.

https://github.com/trapexit/mergerfs/releases/tag/2.28.0

I just released 2.28.0 which has a few significant changes. In Linux 4.20 there were a few features added that I now take advantage of. It can now cache symlinks and readdir (directory listing). When combined with increased `cache.attr` and `cac...

View Post

User Feedback Request: mergerfs

Are there any features people are looking for? Tutorials / howtos they'd like to see?

Some things I'm considering:

* statfs caching: statfs calls are on the expensive side and done regularly but on average the data doesn't change drastically between each call. A cache may help improve performance with certain usage patterns.

* stat caching: less likely to be as useful due to the number of possible values and typical use patterns.

* writeback caching: found in libfuse3. may he...

View Post

mergerfs status update: 2018-12-05

Earlier this year I took a break from mergerfs and took over maintenance of the libretro 4DO core. I cleaned up the code, fixed misc bugs, and made a few quality of life improvements. There is still a lot that could be done to improve performance and compatibility but lack of documentation and complexity of the existing code makes that tough. So I took a break and started working on mergerfs again. As you may have noticed I 2018-12-05 16:07:49 +0000 UTC View Post