[SOLVED] How to safely delete a socket file descriptor used with epoll in a multithreaded program?

Issue

I am trying to solve the epoll race condition problem where an epoll event loop is running and I want it to stop watching a socket file descriptor (FD), but I can't really know after calling epoll_ctl(..., EPOLL_CTL_DEL, ...) if the epoll instance actually deleted it, or if it's handling an event associated with the FD, or if it's just waking up the thread from epoll_wait().

I have googled a little bit, and found a peculiar kernel patch adding a EPOLL_CTL_DISABLE operation (here). Sadly, it requires EPOLLONESHOT, which I don't think is very efficient, since rearming isn't a trivial operation (it requires a search in the epoll's red-black tree).

Thus I was wondering about my own solution. I can obviously just stop the event loop's thread and eventually recreate it later after I'm done dealing with my socket file descriptor, and I can know when the thread is gone by simply doing pthread_join(), but let's not be naive, pthread_join() will make the kernel release the thread's resources and then pthread_create() will allocate the same resources again, which is not ideal as in the EPOLLONESHOT case and will surely take some performance penalty.

Now I'm wondering about something else. If I could somehow manually dispatch an event on the socket that epoll_wait() will wake up to, I can just deploy any blocking synchronization method after it wakes up to be sure that it doesn't go on further. This way, it will go to the next epoll_wait() iteration, where it will realise the socket got removed and no race conditions will happen. To illustrate, something like the following (can really use anything instead of barriers):

/* Obviously, never call the function from
within the event loop, or a deadlock will emerge */
void safe_socket_delete(int sfd) {
  atomic_store(&epoll->should_wait, 1);
  /* This is the function I'm looking for.
  I need to use it to be sure that the event
  loop will go through the should_wait if */
  dispatch_event(sfd, some_event);
  pthread_barrier_wait(&epoll->barrier);
  // we do whatever we want with the socket now
  pthread_barrier_wait(&epoll->barrier);
}

void event_loop() {
  while(1) {
    int events = epoll_wait(epoll->fd, ...);
    if(atomic_load(&epoll->should_wait) == 1) {
      pthread_barrier_wait(&epoll->barrier);
      pthread_barrier_wait(&epoll->barrier);
      continue;
    }
    // handle the events
  }
}

Is there any function like so? If not, how can I deal with the problem efficiently?

Maybe I can approach this differently. Instead of dispatching an event on an existing socket to wake up epoll_wait(), maybe I can create a new event FD using eventfd(), then make it be readable or writeable, add it to the epoll when needed, and tada. Remove it after the fact. I will think a little bit more and maybe try this idea. Of course it comes with 2 tree searches, but better 2 searches when needed than searches when not needed (always need to rearm with EPOLLONESHOT).

Huge optimisation would be to always have one such event file descriptor with an EPOLLET so that I can simulate events on it and get epoll_wait() to wake up. Pros: no tree searches, cons: one more file descriptor per an epoll instance.

Solution

There are several ways to dispatch an event. Since you are using Linux, why don't you use an eventfd https://man7.org/linux/man-pages/man2/eventfd.2.html. Create the eventfd and register it with epoll. When you write to it you will wake up epoll and you can process it like any other file descriptor you need to process.

Answered By - doron