caffe 训练时,出现错误:Check failed: error == cudaSuccess (4 vs. 0) unspecified launch failure

I0415 15:03:37.603461 27311 solver.cpp:42] Solver scaffolding done.
I0415 15:03:37.603549 27311 solver.cpp:247] Solving AlexNet
I0415 15:03:37.603559 27311 solver.cpp:248] Learning Rate Policy: step
I0415 15:03:37.749981 27311 solver.cpp:214] Iteration 0, loss = 5.45141
I0415 15:03:37.750030 27311 solver.cpp:229]     Train net output #0: loss = 5.45141 (* 1 = 5.45141 loss)
I0415 15:03:37.750048 27311 solver.cpp:489] Iteration 0, lr = 0.001
I0415 15:03:38.316994 27311 solver.cpp:214] Iteration 12, loss = 4.23865
I0415 15:03:38.317054 27311 solver.cpp:229]     Train net output #0: loss = 4.23865 (* 1 = 4.23865 loss)
I0415 15:03:38.317068 27311 solver.cpp:489] Iteration 12, lr = 0.001
I0415 15:03:38.920938 27311 solver.cpp:214] Iteration 24, loss = 2.49914
I0415 15:03:38.921000 27311 solver.cpp:229]     Train net output #0: loss = 2.49914 (* 1 = 2.49914 loss)
I0415 15:03:38.921016 27311 solver.cpp:489] Iteration 24, lr = 0.001
I0415 15:03:39.509793 27311 solver.cpp:214] Iteration 36, loss = 3.76504
I0415 15:03:39.509850 27311 solver.cpp:229]     Train net output #0: loss = 3.76504 (* 1 = 3.76504 loss)
I0415 15:03:39.509861 27311 solver.cpp:489] Iteration 36, lr = 0.001
I0415 15:03:40.080806 27311 solver.cpp:214] Iteration 48, loss = 3.74901
I0415 15:03:40.080862 27311 solver.cpp:229]     Train net output #0: loss = 3.74901 (* 1 = 3.74901 loss)
I0415 15:03:40.080878 27311 solver.cpp:489] Iteration 48, lr = 0.001
I0415 15:03:40.643797 27311 solver.cpp:214] Iteration 60, loss = 2.27091
I0415 15:03:40.643849 27311 solver.cpp:229]     Train net output #0: loss = 2.27091 (* 1 = 2.27091 loss)
I0415 15:03:40.643860 27311 solver.cpp:489] Iteration 60, lr = 0.001
I0415 15:03:41.217475 27311 solver.cpp:214] Iteration 72, loss = 2.67078
I0415 15:03:41.217541 27311 solver.cpp:229]     Train net output #0: loss = 2.67078 (* 1 = 2.67078 loss)
I0415 15:03:41.217561 27311 solver.cpp:489] Iteration 72, lr = 0.001
I0415 15:03:41.793390 27311 solver.cpp:214] Iteration 84, loss = 1.77313
I0415 15:03:41.793452 27311 solver.cpp:229]     Train net output #0: loss = 1.77313 (* 1 = 1.77313 loss)
I0415 15:03:41.793468 27311 solver.cpp:489] Iteration 84, lr = 0.001
I0415 15:03:42.362951 27311 solver.cpp:214] Iteration 96, loss = 3.49406
I0415 15:03:42.363004 27311 solver.cpp:229]     Train net output #0: loss = 3.49406 (* 1 = 3.49406 loss)
I0415 15:03:42.363025 27311 solver.cpp:489] Iteration 96, lr = 0.001
I0415 15:03:42.946568 27311 solver.cpp:214] Iteration 108, loss = 2.81601
I0415 15:03:42.946633 27311 solver.cpp:229]     Train net output #0: loss = 2.81601 (* 1 = 2.81601 loss)
I0415 15:03:42.946651 27311 solver.cpp:489] Iteration 108, lr = 0.001
I0415 15:03:43.524155 27311 solver.cpp:214] Iteration 120, loss = 2.85056
I0415 15:03:43.524247 27311 solver.cpp:229]     Train net output #0: loss = 2.85056 (* 1 = 2.85056 loss)
I0415 15:03:43.524265 27311 solver.cpp:489] Iteration 120, lr = 0.001
I0415 15:03:44.100580 27311 solver.cpp:214] Iteration 132, loss = 3.58945
I0415 15:03:44.100646 27311 solver.cpp:229]     Train net output #0: loss = 3.58945 (* 1 = 3.58945 loss)
I0415 15:03:44.100661 27311 solver.cpp:489] Iteration 132, lr = 0.001
F0415 15:03:44.536542 27311 math_functions.cpp:91] Check failed: error == cudaSuccess (4 vs. 0)  unspecified launch failure
*** Check failure stack trace: ***
    @     0x7f01dbd9ddaa  (unknown)
    @     0x7f01dbd9dce4  (unknown)
    @     0x7f01dbd9d6e6  (unknown)
    @     0x7f01dbda0687  (unknown)
    @     0x7f01dc1bb3f5  caffe::caffe_copy<>()
    @     0x7f01dc230232  caffe::BasePrefetchingDataLayer<>::Forward_gpu()
    @     0x7f01dc1d9d6f  caffe::Net<>::ForwardFromTo()
    @     0x7f01dc1da197  caffe::Net<>::ForwardPrefilled()
    @     0x7f01dc20cbe5  caffe::Solver<>::Step()
    @     0x7f01dc20d52f  caffe::Solver<>::Solve()
    @           0x406428  train()
    @           0x404961  main
    @     0x7f01db2afec5  (unknown)
    @           0x404f0d  (unknown)
    @              (nil)  (unknown)
Aborted
[email protected]:~/Downloads/lstm_caffe_master$

---------------------------------------------------------------------------------------------------------------------------------------

怎么破 ???求解答。。。。

时间: 2024-10-09 09:12:37

caffe 训练时,出现错误:Check failed: error == cudaSuccess (4 vs. 0) unspecified launch failure的相关文章

caffe运行错误: im2col.cu:61] Check failed: error == cudaSuccess (8 vs. 0) invalid device function

错误: im2col.cu:61] Check failed: error == cudaSuccess (8 vs. 0)  invalid device function 原因:由于Makefile.config里面只对cuda8.0一下的架构编译cuda程序,所以采用cuda8.0后cuda程序没有被编译 解决:修改Makefile.config中的 CUDA_ARCH 添加 -gencode arch=compute_61,code=sm_61 计算能力可以参考下面链接添加合适的架构参数

Check failed: error == cudaSuccess (10 vs. 0) invalid device ordinal

ubuntu14.04 caffe 运行别人的程序,报错Check failed: error == cudaSuccess (10 vs. 0)  invalid device ordinal.根据网上的说法,GPU内存不够,可是batch大小改为1了还是不行.问了博士师兄才发现,在该行上面有一行,写着using GPUs 1.而我只有一个GPU,如图2,我把solver_48.prototxt中最后一行的device_id改为0,接着运行,见图1,这个错误就没有啦--当然我的又报了别的错..

Run Caffe Meet [Check failed: error == cudaSuccess (8 vs.0) invalid device function]

It is possible relative to CUDA compute cabality , and should find your VGA Driver's compute cabality You can refer to is Blog: http://blog.csdn.net/real_myth/article/details/44308169 And then edit [CUDA_ARCH] in your Makefile.config, for examples, m

Check failed: status == CUBLAS_STATUS_SUCCESS (11 vs. 0) CUBLAS_STATUS_MAPPING_ERROR

I0930 21:23:15.115576 30918 solver.cpp:281] Learning Rate Policy: multistepF0930 21:23:17.263314 31011 math_functions.cu:121] Check failed: status == CUBLAS_STATUS_SUCCESS (11 vs. 0) CUBLAS_STATUS_MAPPING_ERROR*** Check failure stack trace: ***F0930

目标检测faster rcnn error == cudaSuccess (2 vs. 0) out of memory

想尝试 更深更强的网络,或者自己写了一个费显存的层,发现1080 ti的11G显存不够用了,老师报显存不够怎么办? Check failed: error == cudaSuccess (2 vs. 0)  out of memory 调小batch_size的大小. 怎么调呢? 调这个文件: experiments/cfgs/faster_rcnn_end2end.yml EXP_DIR: faster_rcnn_end2endTRAIN: HAS_RPN: True IMS_PER_BATC

[ArchLinux]进行更新时出现错误:GPGME error: 无数据,数据库无效的解决办法/

今天在我进行系统更新pacman -Syu时,出现了如下错误: # pacman -Syu 错误:GPGME error: 无数据 错误:GPGME error: 无数据 错误:GPGME error: 无数据 错误:GPGME error: 无数据 错误:GPGME error: 无数据 :: 正在同步软件包数据库... core 已经是最新版本 0.0 B 0.00B/s 00:00 [----------------------] 0% extra 已经是最新版本 0.0 B 0.00B/

Eclipse启动时出现错误 An internal error occurred during: “Updating indexes”

在Eclipse的workspace下有个.metadata文件夹,Eclipse出现异常的log文件就在这个目录下. 最近出现了这样的错误: 查看日志文件发现:     !ENTRY org.eclipse.core.jobs 4 2 2011-08-03 09:33:32.843     !MESSAGE An internal error occurred during: "Updating indexes".     !STACK 0     java.lang.OutOfMe

Eclipse启动时出现错误 An internal error occurred during: &quot;Updating indexes&quot;

在Eclipse的workspace下有个.metadata文件夹,Eclipse出现异常的log文件就在这个目录下. 最近出现了这样的错误: 查看日志文件发现:     !ENTRY org.eclipse.core.jobs 4 2 2011-08-03 09:33:32.843     !MESSAGE An internal error occurred during: "Updating indexes".     !STACK 0     Java.lang.OutOfMe

解决 Magent 安装编译时出现错误和magent: error while loading shared libraries: libevent-2.0.so.5

一.背景 在 Linux上安装Magent代理服务器软件时,在后面的文章会介绍介绍了Magent解决Memcached服务的负载均衡和容错上.我这边先介绍安装Magent过程中出现的问题. 二.问题 安装Magent过程中出现问题 #tar zxvf magent-0.5.tar.gz #/sbin/ldconfig #sed -i "s#LIBS = -levent#LIBS = -levent -lm#g" Makefile #make 问题一:我在编译make Magent出现问