MPI manager

The MPIManager in NEST manages MPI initialization and configuration checks to ensure proper setup for distributed simulations. It facilitates inter-process communication using MPI functions such as Allgather and Allreduce, and synchronizes processes with barriers. Additionally, it handles node-to-process mappings and enforces correct MPI usage to prevent runtime errors.

class MPIManager : public nest::ManagerInterface

Public Functions

virtual void initialize(const bool) override

Prepare manager for operation.

After this method has completed, the manager should be completely initialized and “ready for action”.

See also

finalize()

Note

Initialization of any given manager may depend on other managers having been initialized before. KernelManager::initialize() is responsible for calling the initialization routines on the specific managers in correct order.

Parameters:

adjust_number_of_threads_or_rng_only – Pass true if calling from kernel_manager::change_number_of_threads() or RandomManager::get_status() to limit operations to those necessary for thread adjustment or switch or re-seeding of RNG.

virtual void finalize(const bool) override

Take down manager after operation.

After this method has completed, all dynamic data structures created by the manager shall be deallocated and containers emptied. Plain variables need not be reset.

See also

initialize()

Note

Finalization of any given manager may depend on other managers not having been finalized yet. KernelManager::finalize() is responsible for calling the initialization routines on the specific managers in correct order, i.e., the opposite order of initialize() calls.

Parameters:

adjust_number_of_threads_or_rng_only – Pass true if calling from kernel_manager::change_number_of_threads() to limit operations to those necessary for thread adjustment.

virtual void set_status(const DictionaryDatum&) override

Set the status of the manager.

See also

get_status()

virtual void get_status(DictionaryDatum&) override

Retrieve the status of the manager.

See also

set_status()

Note

This would ideally be a const function. However, some managers delay the update of internal variables up to the point where they are needed (e.g., before reporting their values to the user, or before simulate is called). An example for this pattern is the call to update_delay_extrema_() right at the beginning of ConnectionManager::get_status().

inline size_t get_num_processes() const

Return the number of processes used during simulation.

This functions returns the number of processes. Since each process has the same number of threads, the total number of threads is given by get_num_threads()*get_num_processes().

inline size_t get_rank() const

Get rank of MPI process.

inline size_t get_process_id_of_vp(const size_t vp) const

Return the process id for a given virtual process.

The real process’ id of a virtual process is defined by the relation: p = (vp mod P), where P is the total number of processes.

void mpi_finalize(int exitcode)

Finalize MPI communication (needs to be separate from MPIManager::finalize when compiled with MUSIC since spikes can arrive and handlers called here)

inline void mpi_abort(int exitcode)

If MPI is available, this method calls MPI_Abort with the exitcode.

void communicate(std::vector<unsigned int> &send_buffer, std::vector<unsigned int> &recv_buffer, std::vector<int> &displacements)

communicate (on-grid) if compiled without MPI

void communicate(std::vector<OffGridSpike> &send_buffer, std::vector<OffGridSpike> &recv_buffer, std::vector<int> &displacements)

communicate (off-grid) if compiled without MPI

void communicate_Allreduce_sum_in_place(double buffer)

Sum across all ranks.

bool equal_cross_ranks(const double value)

Equal across all ranks.

Parameters:

value – value on calling rank

Returns:

true if values across all ranks are equal, false otherwise or if any rank passes -inf as value

inline size_t get_buffer_size_target_data() const

Returns total size of MPI buffer for communication of connections.

inline unsigned int get_send_recv_count_target_data_per_rank() const

Returns size of MPI buffer for connections divided by number of processes.

inline size_t get_buffer_size_spike_data() const

Returns total size of MPI buffer for communication of spikes.

inline unsigned int get_send_recv_count_spike_data_per_rank() const

Returns size of MPI buffer for spikes divided by number of processes.

inline size_t get_send_buffer_size_secondary_events_in_int() const

Returns total size of MPI send buffer for communication of secondary events.

inline size_t get_recv_buffer_size_secondary_events_in_int() const

Returns total size of MPI recv buffer for communication of secondary events.

inline void synchronize()

Ensure all processes have reached the same stage by waiting until all processes have sent a dummy message to process 0.

inline double time_communicate(int num_bytes, int samples = 1000)

Benchmark communication time of different MPI methods.

The methods time_communicate* can be used to benchmark the timing of different MPI communication methods.

inline bool increase_buffer_size_target_data()

Increases the size of the MPI buffer for communication of connections if it needs to be increased.

Returns whether the size was changed.

inline bool adaptive_target_buffers() const

Returns whether MPI buffers for communication of connections are adaptive.

inline void set_recv_counts_secondary_events_in_int_per_rank(const std::vector<int> &recv_counts_in_int_per_rank)

Sets the recvcounts parameter of Alltoallv for communication of secondary events, i.e., the number of elements (in ints) to recv from the corresponding rank.

inline size_t get_recv_count_secondary_events_in_int(const size_t source_rank) const

Returns the recvcounts parameter of Alltoallv for communication of secondary events, i.e., the number of elements (in ints) to recv from source_rank.

inline size_t get_recv_displacement_secondary_events_in_int(const size_t source_rank) const

Returns the rdispls parameter of Alltoallv for communication of secondary events, i.e., the offset in the MPI buffer where elements from source_rank are written.

inline size_t get_send_count_secondary_events_in_int(const size_t target_rank) const

Returns the number of elements (in ints) to be sent to target_rank.

inline size_t get_send_displacement_secondary_events_in_int(const size_t target_rank) const

Returns the send displacement of elements (in ints) to be sent to rank target_rank.

inline size_t get_done_marker_position_in_secondary_events_send_buffer(const size_t target_rank) const

Returns where the done marker is located in the MPI send buffer for target_rank.

inline size_t get_done_marker_position_in_secondary_events_recv_buffer(const size_t source_rank) const

Returns where the done marker is located in the MPI recv buffer for source_rank.

class OffGridSpike

Combined storage of node ID and offset information for off-grid spikes.

Note

This class actually stores the node ID as double internally. This is done so that the user-defined MPI type MPI_OFFGRID_SPIKE, which we use to communicate off-grid spikes, is homogeneous. Otherwise, OpenMPI spends extreme amounts of time on packing and unpacking the data, see #458.

Public Types

typedef unsigned int node_id_external_type

We defined this type explicitly, so that the assert function below always tests the correct type.