出现的问题的描述

序言: 程序需要在每秒中处理10000条左右行情数据,这样的并发量真的是很高所以在一般的通用编程模型下会产生怪异的问题

问题描述: 程序在本地运行大概100M内存的消耗并无大碍程序在windowsserver中运行出现内存跳涨,基本是没几秒钟就跳100K左右内存,而且内存不会降,只会涨.

问题分析: 1.是否是使用的内部zmq消息队列深度堆积导致? 经过排查和api系数调整不是这个原因.

2.是否是因为消息处理慢导致?

注释掉回到函数中自己写的代码,程序居然不在内存暴涨.排除法找问题太容易了. 经过注释的方式排出代码,可能由于回调函数处理过慢,导致消息堆积. 具体是因为服务器上运行服务过多,锁分配的时间片就少,需要的处理时间超时,错过第三方动态库了唯一一次的资源释放的机会,导致内存泄漏.

解决办法: 采用异步方式,对数据处理,尽量减少在回调函数中占用的时间,因为每3秒要调用上万次回调函数,其实压力还是很大的.采用了线程池异步调用使用lambda表达式采用[=] 拷贝方式进行参数传递.尽量在回到函数内减少调用的时间.

顺便贴出来基于c++11的线程池代码

#pragma  once
#include <vector>
#include <queue>
#include <memory>
#include <thread>
#include <mutex>
#include <condition_variable>
#include <future>
#include <functional>
#include <stdexcept>

class ThreadPool {
public:
    ThreadPool(size_t);
    template<class F, class... Args>
    auto enqueue(F&& f, Args&&... args)
        ->std::future<typename std::result_of<F(Args...)>::type>;
    ~ThreadPool();
private:
    // need to keep track of threads so we can join them
    std::vector< std::thread > workers;
    // the task queue
    std::queue< std::function<void()> > tasks;

    // synchronization
    std::mutex queue_mutex;
    std::condition_variable condition;
    bool stop;
};

// the constructor just launches some amount of workers
inline ThreadPool::ThreadPool(size_t threads)
    : stop(false)
{
    for (size_t i = 0; i < threads; ++i)
        workers.emplace_back(
            [this]
    {
        for (;;)
        {
            std::function<void()> task;
            {
                std::unique_lock<std::mutex> lock(this->queue_mutex);
                this->condition.wait(lock,
                    [this] { return this->stop || !this->tasks.empty(); });
                if (this->stop && this->tasks.empty())
                    return;
                task = std::move(this->tasks.front());
                this->tasks.pop();
            }

            task();
        }
    }
    );
}

// add new work item to the pool
template<class F, class... Args>
auto ThreadPool::enqueue(F&& f, Args&&... args)
    -> std::future<typename std::result_of<F(Args...)>::type>
{
    using return_type = typename std::result_of<F(Args...)>::type;

    auto task = std::make_shared< std::packaged_task<return_type()> >(
        std::bind(std::forward<F>(f), std::forward<Args>(args)...)
        );

    std::future<return_type> res = task->get_future();
    {
        std::unique_lock<std::mutex> lock(queue_mutex);

        // don't allow enqueueing after stopping the pool
        if (stop)
            throw std::runtime_error("enqueue on stopped ThreadPool");

        tasks.emplace([task]() { (*task)(); });
    }
    condition.notify_one();
    return res;
}

// the destructor joins all threads
inline ThreadPool::~ThreadPool()
{
    {
        std::unique_lock<std::mutex> lock(queue_mutex);
        stop = true;
    }
    condition.notify_all();
    for (std::thread &worker : workers)
        worker.join();
}
 

// create thread pool with 4 worker threads
ThreadPool pool(4);

// enqueue and store future
auto result = pool.enqueue([](int answer) { return answer; }, 42);

// get result from future
std::cout << result.get() << std::endl;