Agfdhyk

TensorFlow 1.11 fails to build with CUDA 8. I tried opening an issue on github (Issue opened on Github #23256 [https://github.com/tensorflow/tensorflow/issues/23256]) but the tensorflow team's response is to just upgrade CUDA to 9 or downgrade Tensorflow to 1.10, which isn't an option for me. Trying to find a way to get TF1.11 to work with CUDA 8.

Attempting to build a docker container with TF 1.11 and CUDA 8 on an GeForce 1060 3GB GPU. An error keeps occurring in the build. Github Issue 22729 (#22729) was looked at but the work around didn't work for TF 1.11 and that's what is needed. The docker file is also below. Any help you can provide would be greatly appreciated.

System information

OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04
TensorFlow installed from (source or binary): Source

TensorFlow version: TF 1.11

Python version: 2.7

Installed using virtualenv? pip? conda?: Docker

Bazel version (if compiling from source): 0.15.0

GCC/Compiler version (if compiling from source): 7.3.0

CUDA/cuDNN version: 8.0/7

GPU model and memory: GeForce GTX 1060 3GB

Provide the exact sequence of commands / steps that you executed before running into the problem
sudo docker build --no-cache . -f Dockerfile.tf-1.11-py27-gpu.txt -t tf-1.11-py27-gpu

Thank you,
Kyle

Dockerfile.tf-1.11-py27-gpu

FROM nvidia/cuda:8.0-cudnn7-devel-ubuntu16.04



LABEL maintainer="Craig Citro <craigcitro@google.com>; Modified for Cuda 8 by Jack Harris"



RUN apt-get update && apt-get install -y --allow-downgrades --allow-change-held-packages --no-install-recommends 

    build-essential 

    cuda-command-line-tools-8-0 

    cuda-cublas-dev-8-0 

    cuda-cudart-dev-8-0 

    cuda-cufft-dev-8-0 

    cuda-curand-dev-8-0 

    cuda-cusolver-dev-8-0 

    cuda-cusparse-dev-8-0 

    curl 

    git 

    libcudnn7=7.2.1.38-1+cuda8.0 

    libcudnn7-dev=7.2.1.38-1+cuda8.0 

    libnccl2=2.2.13-1+cuda8.0 

    libnccl-dev=2.2.13-1+cuda8.0 

    libcurl3-dev 

    libfreetype6-dev 

    libhdf5-serial-dev 

    libpng12-dev 

    libzmq3-dev 

    pkg-config 

    python-dev 

    rsync 

    software-properties-common 

    unzip 

    zip 

    zlib1g-dev 

    wget 

    && 

rm -rf /var/lib/apt/lists/* && 

find /usr/local/cuda-8.0/lib64/ -type f -name 'lib*_static.a' -not -name 'libcudart_static.a' -delete && 

rm -f /usr/lib/x86_64-linux-gnu/libcudnn_static_v7.a



RUN apt-get update && 

    apt-get install nvinfer-runtime-trt-repo-ubuntu1604-4.0.1-ga-cuda8.0 && 

    apt-get update && 

    apt-get install libnvinfer4=4.1.2-1+cuda8.0 && 

    apt-get install libnvinfer-dev=4.1.2-1+cuda8.0



# Link NCCL libray and header where the build script expects them.

RUN mkdir /usr/local/cuda-8.0/lib &&  

ln -s /usr/lib/x86_64-linux-gnu/libnccl.so.2 /usr/local/cuda/lib/libnccl.so.2 && 

ln -s /usr/include/nccl.h /usr/local/cuda/include/nccl.h



# TODO(tobyboyd): Remove after license is excluded from BUILD file.

#RUN gunzip /usr/share/doc/libnccl2/NCCL-SLA.txt.gz && 

#    cp /usr/share/doc/libnccl2/NCCL-SLA.txt /usr/local/cuda/



# Add External Mount Points

RUN mkdir -p /external_lib

RUN mkdir -p /external_bin



RUN curl -fSsL -O https://bootstrap.pypa.io/get-pip.py && 

    python get-pip.py && 

    rm get-pip.py



RUN pip --no-cache-dir install 

    ipykernel 

    jupyter 

    keras_applications==1.0.5 

    keras_preprocessing==1.0.3 

    matplotlib 

    numpy 

    pandas 

    scipy 

    sklearn 

    mock  

    && 

python -m ipykernel.kernelspec



# Set up our notebook config.

#COPY jupyter_notebook_config.py /root/.jupyter/



# Jupyter has issues with being run directly:

#   https://github.com/ipython/ipython/issues/7062

# We just add a little wrapper script.

# COPY run_jupyter.sh /



# Set up Bazel.



# Running bazel inside a `docker build` command causes trouble, cf:

#   https://github.com/bazelbuild/bazel/issues/134

# The easiest solution is to set up a bazelrc file forcing --batch.

RUN echo "startup --batch" >>/etc/bazel.bazelrc

# Similarly, we need to workaround sandboxing issues:

#   https://github.com/bazelbuild/bazel/issues/418

RUN echo "build --spawn_strategy=standalone --genrule_strategy=standalone" 

    >>/etc/bazel.bazelrc

# Install the most recent bazel release.

ENV BAZEL_VERSION 0.15.0

WORKDIR /

RUN mkdir /bazel && 

    cd /bazel && 

    curl -H "User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36     (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36" -fSsL -O https://github.com/bazelbuild/bazel/releases/download/$BAZEL_VERSION/bazel-$BAZEL_VERSION-installer-linux-x86_64.sh && 

curl -H "User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36" -fSsL -o /bazel/LICENSE.txt https://raw.githubusercontent.com/bazelbuild/bazel/master/LICENSE && 

chmod +x bazel-*.sh && 

./bazel-$BAZEL_VERSION-installer-linux-x86_64.sh && 

cd / && 

rm -f /bazel/bazel-$BAZEL_VERSION-installer-linux-x86_64.sh



# Download and build TensorFlow.

RUN git clone http://github.com/tensorflow/tensorflow --branch r1.11 --depth=1



WORKDIR /tensorflow



RUN sed -i 's/^#if TF_HAS_.*$/#if !defined(__NVCC__)/g' tensorflow/core/platform/macros.h



ENV TF_NCCL_VERSION=2



#RUN /bin/echo -e "/usr/bin/pythonnnnnnnnnnnnnnnnnnnnnyn8.0n/usr/local/cudan7.0n/usr/local/cudannnnnnnnn-march=nativennn" | ./configure

RUN /bin/echo -e "/usr/bin/pythonnnnnnnnnnnnnnnnnnnnnnnyn8.0n/usr/local/cudan7.0n/usr/local/cudannnnnnnnnnnnn-march=nativennn" | ./configure

#RUN /bin/echo -e "nnnnnnnnnnnnnnnnnnnnnnn-march=nativennn" | ./configure



# Configure the build for our CUDA configuration.

ENV CI_BUILD_PYTHON python

ENV PATH /external_bin:$PATH

ENV LD_LIBRARY_PATH /external_lib:/usr/local/cuda/extras/CUPTI/lib64:$LD_LIBRARY_PATH

ENV TF_NEED_CUDA 1

ENV TF_NEED_TENSORRT 1

ENV TF_CUDA_COMPUTE_CAPABILITIES=3.0,3.5,5.2,6.0,6.1

ENV TF_CUDA_VERSION=8.0

ENV TF_CUDNN_VERSION=7



# https://github.com/tensorflow/tensorflow/issues/17801

RUN ln -s /usr/local/cuda/lib64/stubs/libcuda.so /usr/local/cuda/lib64/stubs/libcuda.so.1 && 

ln -s /usr/local/cuda/nvvm/libdevice/libdevice.compute_50.10.bc /usr/local/cuda/nvvm/libdevice/libdevice.10.bc && 

LD_LIBRARY_PATH=/usr/local/cuda/lib64/stubs:${LD_LIBRARY_PATH} 

tensorflow/tools/ci_build/builds/configured GPU 

bazel build -c opt --copt=-mavx --config=cuda 

--cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0" 

    tensorflow/tools/pip_package/build_pip_package && 

    rm /usr/local/cuda/lib64/stubs/libcuda.so.1 



RUN bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/pip 



RUN pip --no-cache-dir install --upgrade /tmp/pip/tensorflow-*.whl && 

rm -rf /tmp/pip && 

rm -rf /root/.cache

# Clean up pip wheel and Bazel cache when done.



WORKDIR /root



# TensorBoard

EXPOSE 6006

# IPython

EXPOSE 8888



CMD [ "/bin/bash" ]

tf11cuda8.log - Log attached to github issue (too long to post here)