Hi,

By any chance has anyone linked against libnotify so as to monitor certain changes in a directory and subdirectories, and act according to event?
I am thinking in combining this with fossil [1] to automatically document changes in /etc

My main concern is that (if I undestand it correctly), inotify requires a file descriptor for every object watched, and if a directory has many files this could probably have an impact on performance, or do you know if just by using the directory I want to monitor, everything within it will also be watched?

Also, I have never linked before against a Linux library and inotify does not seem particularly easy because of the file descriptors, so if someone has linked against this library and has a sample code, I would very much appreciate if you could share it.

Regards, Hugo

[1] www.fossil-scm.org

Here is a first attempt. I'm sure there are much better ways to do this but this should at least get you going in the right direction.

Some of the key things to know are the undocumented (at least not documented on the web site) modules that are available. One of them contains the Linux inotify header. If you've downloaded the zip file from the website take a look at the src/druntime/src directory for these modules.

I took the example from http://stackoverflow.com/questions/4062806/inotify-how-to-use-it-linux and made some changes to get it to compile and run in D.

import core.sys.linux.sys.inotify;
import core.stdc.stdlib : malloc;
import core.sys.posix.unistd : read;

import std.stdio : writeln;
import std.string: toStringz;

enum PATH_MAX = 256;

void main()
{
    string filename="aaa";

    int inotfd = inotify_init();
    int watch_desc = inotify_add_watch(inotfd, toStringz(filename), IN_MODIFY);

    size_t bufsiz = inotify_event.sizeof + PATH_MAX + 1;
    inotify_event* event = cast(inotify_event *) malloc(bufsiz);

    /* wait for an event to occur */
    read(inotfd, event, event.sizeof);

    /* process event struct here */
    writeln("Received inotify event.");
}

As far as monitoring a directory, you can do that with inotify the same way you would a file. See http://www.ibm.com/developerworks/linux/library/l-ubuntu-inotify/index.html for more a C example (Listing 1).

On Fri, 27 Dec 2013 03:23:00 +0000, David Eagen wrote:
> Here is a first attempt. I'm sure there are much better ways to do
> this but this should at least get you going in the right direction.
> ...

Thanks!

BTW, it it a requirement to use malloc, and if so, when would I need to free the memory allocated by it?

On Friday, 27 December 2013 at 03:39:58 UTC, Hugo Florentino wrote:
> BTW, it it a requirement to use malloc, and if so, when would I need to free the memory allocated by it?

Inotify is expecting you to read into a buffer. You could allocate that buffer from the GC with core.memory.malloc() instead.

You want to free the memory once you finish processing the event.

On 12/27/13 04:23, David Eagen wrote:
> void main()
> {
>     string filename="aaa";
> 
>     int inotfd = inotify_init();
>     int watch_desc = inotify_add_watch(inotfd, toStringz(filename), IN_MODIFY);
> 
>     size_t bufsiz = inotify_event.sizeof + PATH_MAX + 1;
>     inotify_event* event = cast(inotify_event *) malloc(bufsiz);
> 
>     /* wait for an event to occur */
>     read(inotfd, event, event.sizeof);

You probably meant

   read(inotfd, event, (*event).sizeof);

but in this case the inotify_event structure contains an optional trailing buffer, so it should be

   read(inotfd, event, bufsiz);

artur

On Friday, 27 December 2013 at 03:39:58 UTC, Hugo Florentino wrote:
> BTW, it it a requirement to use malloc, and if so, when would I need to free the memory allocated by it?

I use a static ubyte array.

I've been using inotify quite a bit and found it to be very good but there are a few things to keep in mind though. First it can block further program execution when watching files. To avoid this use the select function in 'core.sys.posix.sys.select'. Second, if the file doesn't exist or is moved once watched, the inotifiy instance or the watch descriptor will be invalid and need to be re-initialised. Tip: moving can occur if edited with a text editor. As i found out trying to test inotify by changing a file using vim! That was debug pain!

Here is a snippet of code to show how i used it. It's not complete but it shows you how to use the select function and how all the bits fit together. I used an infinite loop because this was part of a daemon that runs forever so you may want to handle that better.

/**
 * Module.
 */
module common.file.watcher.inotifyengine;

/**
 * Imports.
 */
import core.sys.linux.sys.inotify;
import core.sys.posix.sys.select;
import core.sys.posix.unistd;
import common.file.logger;
import common.file.watcher.engine;
import std.string;

/**
 * A class to watch for changes in a file.
 *
 * Uses the linux inotify subsystem.
 */
class INotifyEngine : Engine
{
	/**
	 * The event buffer length.
	 *
	 * This gives us for 1024 events which should be more than enough.
	 */
	private enum eventBufferLength = (inotify_event.sizeof + 16) * 1024;

	/**
	 * The notification flags.
	 *
	 * These are what inotify uses to descriminate on which events to nofify us about.
	 */
	private enum notificationFlags  = IN_MODIFY | IN_ATTRIB | IN_DELETE_SELF | IN_MOVE_SELF | IN_IGNORED;

	/**
	 * The inotify instance.
	 */
	private int _inotifyInstance;

	/**
	 * The watch descriptor.
	 */
	private int _watchDescriptor;

	/**
	 * The file descriptor set for the select call.
	 */
	private fd_set _fileDescriptorSet;

	/**
	 * The timeout for the select call.
	 */
	private timeval _timeout;

	/**
	 * Constructor.
	 *
	 * Params:
	 *     logger = The logger object used to log messages.
	 *
	 * See_Also:
	 *     Engine
	 */
	public this(Logger logger)
	{
		super(logger);
	}

	/**
	 * Initialize inotify.
	 *
	 * Throws:
	 *     Exception if inotify fails to initialize.
	 */
	private void initInotify()
	{
		this._inotifyInstance = inotify_init();

		if (this._inotifyInstance < 0)
		{
			this._logger.error("Inotify failed to initialize.");
		}

		this._watchDescriptor = inotify_add_watch(this._inotifyInstance, cast(char*)this._lastFileName.toStringz(), notificationFlags);
	}

	/**
	 * Stop inotify.
	 */
	private void closeInotify()
	{
		inotify_rm_watch(this._inotifyInstance, this._watchDescriptor);
		close(this._inotifyInstance);
	}

	/**
	 * Change the watcher if the file name changes.
	 */
	private void checkFileNameChange()
	{
		string currentFileName = this.getFileName();

		if (currentFileName != this._lastFileName)
		{
			this._logger.warning("Watched file name changed to '%s'", currentFileName);

			this.notify(this._lastFileName);
			this._lastFileName = currentFileName;
			this.notify(this._lastFileName);

			this.closeInotify();
			this.initInotify();

			this._logger.info("Starting to watch '%s' for alerts (%d, %d)", this._lastFileName, this._inotifyInstance, this._watchDescriptor);
		}
	}

	/**
	 * Retry watching if there was a problem e.g. the file doesn't exist yet.
	 */
	private void checkWatchStatus()
	{
		if (this._inotifyInstance == -1 || this._watchDescriptor == -1)
		{
			this._logger.error("Failed watching '%s' for alerts, retrying...", this._lastFileName);

			this.closeInotify();
			this.initInotify();

			this._logger.info("Starting to watch '%s' for alerts (%d, %d)", this._lastFileName, this._inotifyInstance, this._watchDescriptor);
		}
	}

	/**
	 * Check the file for any changes.
	 *
	 * If changes occur then execute the action if one has been assigned.
	 * We are using the select call to perform non blocking event handling waiting for inotify.
	 * If inotify detects a change in the file, select returns immediately.
	 *
	 * See_Also:
	 *     IEngine
	 */
	public void start()
	{
		this._lastFileName = this.getFileName();

		this.notify(this._lastFileName);

		this.initInotify();

		this._logger.info("Starting to watch '%s' for alerts (%d, %d)", this._lastFileName, this._inotifyInstance, this._watchDescriptor);

		ubyte[eventBufferLength] eventBuffer;
		void* eventPointer;

		while (true)
		{
			FD_ZERO(&this._fileDescriptorSet);
			FD_SET(this._inotifyInstance, &this._fileDescriptorSet);

			this._timeout.tv_usec = 0;
			this._timeout.tv_sec  = this._interval;

			if (select(FD_SETSIZE, &this._fileDescriptorSet, null, null, &this._timeout))
			{
				auto bytesRead = read(this._inotifyInstance, eventBuffer.ptr, eventBuffer.length);

				for (eventPointer = eventBuffer.ptr; eventPointer < eventBuffer.ptr + bytesRead; null)
				{
					inotify_event* event = cast(inotify_event*)eventPointer;

					if (event.mask & IN_MODIFY || event.mask & IN_ATTRIB)
					{
						this.notify(this._lastFileName);
					}

					if (event.mask & IN_DELETE_SELF || event.mask & IN_MOVE_SELF || event.mask & IN_IGNORED)
					{
						this.notify(this._lastFileName);

						this.closeInotify();
						this.initInotify();

						this._logger.warning("Alert log moved or deleted, re-initialized to watch '%s' again (%d, %d)", this._lastFileName, this._inotifyInstance, this._watchDescriptor);
					}

					eventPointer += inotify_event.sizeof + event.len;
				}
			}

			this.checkFileNameChange();
			this.checkWatchStatus();
		}
	}

	/**
	 * Destructor.
	 */
	~this()
	{
		this.closeInotify();
	}
}

On Friday, 27 December 2013 at 10:56:55 UTC, Artur Skawina wrote:
>
> You probably meant
>
>    read(inotfd, event, (*event).sizeof);
>
> but in this case the inotify_event structure contains an optional
> trailing buffer, so it should be
>
>    read(inotfd, event, bufsiz);
>
> artur

Yes, thanks for the correction.

I had trouble getting the file name from the event. I think it's because the inotify module has name defined as char[0]. So I did this, which prints the name by using the extra data beyond the inotify_event struct itself:


    void* buf = GC.malloc(bufsiz);

    /* wait for an event to occur */
    size_t readlen = read(inotfd, buf, bufsiz);
    inotify_event* event = cast(inotify_event*) (buf);

    /* process event struct here */
    writeln("Received inotify event:");
    writeln("Bytes read: ", readlen);
    writeln("Length: ", event.len);
    writeln("Name:", cast(char[])(buf[event.len..readlen]));

On 12/27/13 14:28, David Eagen wrote:
> I had trouble getting the file name from the event. I think it's because the inotify module has name defined as char[0]. So I did this, which prints the name by using the extra data beyond the inotify_event struct itself:
> 
> 
>     void* buf = GC.malloc(bufsiz);
> 
>     /* wait for an event to occur */
>     size_t readlen = read(inotfd, buf, bufsiz);
>     inotify_event* event = cast(inotify_event*) (buf);
> 
>     /* process event struct here */
>     writeln("Received inotify event:");
>     writeln("Bytes read: ", readlen);
>     writeln("Length: ", event.len);
>     writeln("Name:", cast(char[])(buf[event.len..readlen]));

   writeln("Name:", (cast(char*)&event.name)[0..event.len-1]); // 2do: strip any extra trailing \0s.

It's probably easier (and safer) if you do it like this:

   struct MyInotifyEvent(size_t BS) {
      inotify_event event;
      char[BS] buffer;
      alias event this;
   }
   [...]
   enum bufsiz = inotify_event.sizeof + PATH_MAX + 1;
   auto event = cast(MyInotifyEvent!bufsiz*)malloc(bufsiz);
   [...]
   writeln("Name:", event.buffer[0..event.len-1]); // 2do: strip any extra trailing \0s.

artur

On Fri, 27 Dec 2013 12:56:25 +0000, Gary Willoughby wrote:
> On Friday, 27 December 2013 at 03:39:58 UTC, Hugo Florentino wrote:
>> BTW, it it a requirement to use malloc, and if so, when would I need to free the memory allocated by it?
>
> I use a static ubyte array.
>
> I've been using inotify quite a bit and found it to be very good but
> there are a few things to keep in mind though. First it can block
> further program execution when watching files. To avoid this use the
> select function in 'core.sys.posix.sys.select'.

Thanks. Have you ever tried using epoll?
I have read that for asynchronous things it works much better than select.

On 12/27/13 15:13, Artur Skawina wrote:
>    struct MyInotifyEvent(size_t BS) {
>       inotify_event event;
>       char[BS] buffer;
>       alias event this;
>    }
>    [...]
>    enum bufsiz = inotify_event.sizeof + PATH_MAX + 1;
>    auto event = cast(MyInotifyEvent!bufsiz*)malloc(bufsiz);
>    [...]
>    writeln("Name:", event.buffer[0..event.len-1]); // 2do: strip any extra trailing \0s.

Well, that could get awkward, as the inotify api doesn't really support disabling event batching. A saner design would be something like the code below.

artur

struct INotify {
   Fd fd;

   import std.exception;
   import core.sys.posix.unistd;

   this(cint flags) {
      fd = inotify_init(flags);
      errnoEnforce(fd!=-1);
   }
   ~this() {
      close(fd);
   }

   Wd add(const char* name, uint mask) { return inotify_add_watch(fd, name, mask); }
   Fd remove(Wd wd)                    { return inotify_rm_watch(fd, wd); }

   bool get(CB)(CB cb) {
      void[4*1024] buffer = void;

      auto got = read(fd, &buffer, buffer.sizeof);

      if (got<=inotify_event.sizeof)
         return false;

      size_t off = 0;

      while (off<=got-inotify_event.sizeof) {
         auto evp = cast(inotify_event*)&buffer[off];
         auto namebuf = evp.len ? (cast(char*)&evp.name)[0..evp.len-1] : null;

         while (namebuf.length && !namebuf[namebuf.length-1])
            --namebuf.length;

         cb(evp, namebuf);

         off += inotify_event.sizeof + evp.len;
      }

      assert(off==got);
      return true;
   }
}

void main(string argv[]) {
   import std.string;
   auto ntf = INotify(0);

   foreach (arg; argv[1..$])
      ntf.add(toStringz(arg), IN_MODIFY);

   void myCallback(/*scope*/ const inotify_event* ev, /*scope*/ char[] name) {
      /* Don't even think about escaping any of the args. Dup if necessary. */
      import std.stdio;
      writeln(*ev, " \"", name, "\"");
   }

   while (1)
      ntf.get(&myCallback);
}


// libc i/f follows:

alias uint32_t = uint;
alias cint = int;
alias Fd = cint;

extern(C) {
   Fd inotify_init();
   Fd inotify_init1(cint flags);
   Wd inotify_add_watch(Fd fd, const char* pathname, uint mask);
   Fd inotify_rm_watch(Fd fd, Wd wd);
}

Fd inotify_init(cint flags) { return inotify_init1(flags); }

enum IN_MODIFY = 0x00000002; /* DEFINEME */

struct Wd { Fd fd; alias fd this; }

struct inotify_event {
   Wd       wd;       /* Watch descriptor */
   uint32_t mask;     /* Mask of events */
   uint32_t cookie;   /* Unique cookie associating related events (for rename(2)) */
   uint32_t len;      /* Size of name field */
   char     name[0];   /* Optional null-terminated name */

   void toString(DG, FT)(scope DG sink, FT fmt) const @trusted {
      import std.format;
      foreach (I, E; this.tupleof)
         static if (I<this.tupleof.length-1)
            formatValue(sink, E, fmt), sink(" ");
   }
}