Skip to content

Commit 8618f07

Browse files
jetsonhacksjetsonhacks
committed
Add Incremental Bazel Build fix
1 parent 433bae2 commit 8618f07

File tree

289 files changed

+1348
-1
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

289 files changed

+1348
-1
lines changed

‎README.md‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
# installTensorFlowTX1
2-
December 28, 2016
2+
December 28, 2016
3+
Last modified Jan 12, 2017
34

45
Install TensorFlow r0.11 on NVIDIA Jetson TX1 Development Kit
56

@@ -51,3 +52,7 @@ $ cd $HOME/tensorflow
5152

5253
$ time python tensorflow/models/image/mnist/convolutional.py
5354

55+
#### Build Issues
56+
57+
For various reasons, the build may fail. The 'debug' folder contains instructions on how to resume an incremental build.
58+

‎debug/README.md‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
If you're looking at this document, something has probably gone wrong with the TensorFlow build. This assumes that the script 'buildTensorFlow.sh' failed to finish properly.
2+
3+
Here's a couple of tips:
4+
5+
STEP 1
6+
7+
First, as of this writing (Jan 12, 2017) running buildTensorFlow.sh a second time causes an error stating that the GPU processing is not enabled, though the CUDA flag is true in the bazel build line. This is due to an issue in the file:
8+
9+
tensorflow/third_party/gpus/cuda_configure.bzl
10+
11+
Where the rule:
12+
13+
cuda_configure = repository_rule(
14+
implementation = _cuda_autoconf_impl,
15+
local = True,
16+
)
17+
18+
forces Bazel to always rebuild the CUDA configuration, which in turn foobars the incremental build. In order to fix that, run the patch:
19+
20+
$ ./patchCUDAConfig.sh
21+
22+
If you attempted to run buildTensorFlow.sh before patching, you will need to replace the folder local_config_cuda in the folder:
23+
24+
/home/ubuntu/.cache/bazel/_bazel_ubuntu/<id>/external
25+
26+
with the one in this directory. Ideally, you would save the folder after it is generatred, and before it gets overwritten. Note: This may work, or it may not. There is a 'bin' directory which has symbolic pointers which may not match your system. If that is the case, you will have to run 'setTensorFlowEV.sh' and 'buildTensorFlow.sh' again and rebuild everything.
27+
28+
STEP 2
29+
30+
$ ./exportEV.sh
31+
32+
Which exports the needed TF CUDA Environment variables
33+
34+
STEP 3
35+
36+
$ ./buildTensorFlow.sh
37+
38+
This is a more verbose version of the original buildTensorFlow.sh in the parent directory. A file named 'explain.txt' is generated in the $HOME/tensorflow directory.
39+
40+
This should allow for incremental compilation at least, and show you where the build is failing.
41+
42+
43+
44+

‎debug/buildLog.txt‎

Lines changed: 177 additions & 0 deletions
Large diffs are not rendered by default.

‎debug/buildTensorFlow.sh‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
#!/bin/bash
2+
# NVIDIA Jetson TX1
3+
# TensorFlow Installation
4+
# Build Tensorflow
5+
cd $HOME/tensorflow
6+
bazel build -c opt --local_resources 3072,4.0,1.0 --verbose_failures --explain explain.txt --verbose_explanations --config=cuda //tensorflow/tools/pip_package:build_pip_package
7+

‎debug/exportEV.sh‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
#!/bin/bash
2+
# NVIDIA Jetson TX1
3+
# TensorFlow Installation
4+
# Export TensorFlow GPU environment variables
5+
export TF_NEED_CUDA=1
6+
export TF_CUDA_VERSION=8.0
7+
export CUDA_TOOLKIT_PATH=/usr/local/cuda
8+
export TF_CUDNN_VERSION=5.1.5
9+
export CUDNN_INSTALL_PATH=/usr/lib/aarch64-linux-gnu/
10+
export TF_CUDA_COMPUTE_CAPABILITIES=5.3
11+

‎debug/local_config_cuda/WORKSPACE‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# DO NOT EDIT: automatically generated WORKSPACE file for cuda_configure rule
2+
workspace(name = "local_config_cuda")

‎debug/local_config_cuda/crosstool/BUILD‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
licenses(["restricted"])
2+
3+
package(default_visibility = ["//visibility:public"])
4+
5+
cc_toolchain_suite(
6+
name = "toolchain",
7+
toolchains = {
8+
"local|compiler": ":cc-compiler-local",
9+
"darwin|compiler": ":cc-compiler-darwin",
10+
},
11+
)
12+
13+
cc_toolchain(
14+
name = "cc-compiler-local",
15+
all_files = ":empty",
16+
compiler_files = ":empty",
17+
cpu = "local",
18+
dwp_files = ":empty",
19+
dynamic_runtime_libs = [":empty"],
20+
linker_files = ":empty",
21+
objcopy_files = ":empty",
22+
static_runtime_libs = [":empty"],
23+
strip_files = ":empty",
24+
supports_param_files = 0,
25+
)
26+
27+
cc_toolchain(
28+
name = "cc-compiler-darwin",
29+
all_files = ":empty",
30+
compiler_files = ":empty",
31+
cpu = "darwin",
32+
dwp_files = ":empty",
33+
dynamic_runtime_libs = [":empty"],
34+
linker_files = ":empty",
35+
objcopy_files = ":empty",
36+
static_runtime_libs = [":empty"],
37+
strip_files = ":empty",
38+
supports_param_files = 0,
39+
)
40+
41+
filegroup(
42+
name = "empty",
43+
srcs = [],
44+
)

‎debug/local_config_cuda/crosstool/CROSSTOOL‎

Lines changed: 255 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,255 @@
1+
major_version: "local"
2+
minor_version: ""
3+
default_target_cpu: "same_as_host"
4+
5+
default_toolchain {
6+
cpu: "k8"
7+
toolchain_identifier: "local_linux"
8+
}
9+
default_toolchain {
10+
cpu: "piii"
11+
toolchain_identifier: "local_linux"
12+
}
13+
default_toolchain {
14+
cpu: "arm"
15+
toolchain_identifier: "local_linux"
16+
}
17+
default_toolchain {
18+
cpu: "darwin"
19+
toolchain_identifier: "local_darwin"
20+
}
21+
default_toolchain {
22+
cpu: "ppc"
23+
toolchain_identifier: "local_linux"
24+
}
25+
26+
toolchain {
27+
abi_version: "local"
28+
abi_libc_version: "local"
29+
builtin_sysroot: ""
30+
compiler: "compiler"
31+
host_system_name: "local"
32+
needsPic: true
33+
supports_gold_linker: false
34+
supports_incremental_linker: false
35+
supports_fission: false
36+
supports_interface_shared_objects: false
37+
supports_normalizing_ar: false
38+
supports_start_end_lib: false
39+
supports_thin_archives: false
40+
target_libc: "local"
41+
target_cpu: "local"
42+
target_system_name: "local"
43+
toolchain_identifier: "local_linux"
44+
45+
tool_path { name: "ar" path: "/usr/bin/ar" }
46+
tool_path { name: "compat-ld" path: "/usr/bin/ld" }
47+
tool_path { name: "cpp" path: "/usr/bin/cpp" }
48+
tool_path { name: "dwp" path: "/usr/bin/dwp" }
49+
# As part of the TensorFlow release, we place some cuda-related compilation
50+
# files in @local_config_cuda//crosstool/clang/bin, and this relative
51+
# path, combined with the rest of our Bazel configuration causes our
52+
# compilation to use those files.
53+
tool_path { name: "gcc" path: "clang/bin/crosstool_wrapper_driver_is_not_gcc" }
54+
# Use "-std=c++11" for nvcc. For consistency, force both the host compiler
55+
# and the device compiler to use "-std=c++11".
56+
cxx_flag: "-std=c++11"
57+
linker_flag: "-lstdc++"
58+
linker_flag: "-B/usr/bin/"
59+
60+
cxx_builtin_include_directory: "/usr/include/c++/5"
61+
cxx_builtin_include_directory: "/usr/include/aarch64-linux-gnu/c++/5"
62+
cxx_builtin_include_directory: "/usr/include/c++/5/backward"
63+
cxx_builtin_include_directory: "/usr/lib/gcc/aarch64-linux-gnu/5/include"
64+
cxx_builtin_include_directory: "/usr/local/include"
65+
cxx_builtin_include_directory: "/usr/lib/gcc/aarch64-linux-gnu/5/include-fixed"
66+
cxx_builtin_include_directory: "/usr/include/aarch64-linux-gnu"
67+
cxx_builtin_include_directory: "/usr/include"
68+
tool_path { name: "gcov" path: "/usr/bin/gcov" }
69+
70+
# C(++) compiles invoke the compiler (as that is the one knowing where
71+
# to find libraries), but we provide LD so other rules can invoke the linker.
72+
tool_path { name: "ld" path: "/usr/bin/ld" }
73+
74+
tool_path { name: "nm" path: "/usr/bin/nm" }
75+
tool_path { name: "objcopy" path: "/usr/bin/objcopy" }
76+
objcopy_embed_flag: "-I"
77+
objcopy_embed_flag: "binary"
78+
tool_path { name: "objdump" path: "/usr/bin/objdump" }
79+
tool_path { name: "strip" path: "/usr/bin/strip" }
80+
81+
# Anticipated future default.
82+
unfiltered_cxx_flag: "-no-canonical-prefixes"
83+
84+
# Make C++ compilation deterministic. Use linkstamping instead of these
85+
# compiler symbols.
86+
unfiltered_cxx_flag: "-Wno-builtin-macro-redefined"
87+
unfiltered_cxx_flag: "-D__DATE__=\"redacted\""
88+
unfiltered_cxx_flag: "-D__TIMESTAMP__=\"redacted\""
89+
unfiltered_cxx_flag: "-D__TIME__=\"redacted\""
90+
91+
# Security hardening on by default.
92+
# Conservative choice; -D_FORTIFY_SOURCE=2 may be unsafe in some cases.
93+
# We need to undef it before redefining it as some distributions now have
94+
# it enabled by default.
95+
compiler_flag: "-U_FORTIFY_SOURCE"
96+
compiler_flag: "-D_FORTIFY_SOURCE=1"
97+
compiler_flag: "-fstack-protector"
98+
compiler_flag: "-fPIE"
99+
linker_flag: "-pie"
100+
linker_flag: "-Wl,-z,relro,-z,now"
101+
102+
# Enable coloring even if there's no attached terminal. Bazel removes the
103+
# escape sequences if --nocolor is specified. This isn't supported by gcc
104+
# on Ubuntu 14.04.
105+
# compiler_flag: "-fcolor-diagnostics"
106+
107+
# All warnings are enabled. Maybe enable -Werror as well?
108+
compiler_flag: "-Wall"
109+
# Enable a few more warnings that aren't part of -Wall.
110+
compiler_flag: "-Wunused-but-set-parameter"
111+
# But disable some that are problematic.
112+
compiler_flag: "-Wno-free-nonheap-object" # has false positives
113+
114+
# Keep stack frames for debugging, even in opt mode.
115+
compiler_flag: "-fno-omit-frame-pointer"
116+
117+
# Anticipated future default.
118+
linker_flag: "-no-canonical-prefixes"
119+
unfiltered_cxx_flag: "-fno-canonical-system-headers"
120+
# Have gcc return the exit code from ld.
121+
linker_flag: "-pass-exit-codes"
122+
# Stamp the binary with a unique identifier.
123+
linker_flag: "-Wl,--build-id=md5"
124+
linker_flag: "-Wl,--hash-style=gnu"
125+
# Gold linker only? Can we enable this by default?
126+
# linker_flag: "-Wl,--warn-execstack"
127+
# linker_flag: "-Wl,--detect-odr-violations"
128+
129+
# Include directory for cuda headers.
130+
cxx_builtin_include_directory: "/usr/local/cuda-8.0/include"
131+
132+
compilation_mode_flags {
133+
mode: DBG
134+
# Enable debug symbols.
135+
compiler_flag: "-g"
136+
}
137+
compilation_mode_flags {
138+
mode: OPT
139+
140+
# No debug symbols.
141+
# Maybe we should enable https://gcc.gnu.org/wiki/DebugFission for opt or
142+
# even generally? However, that can't happen here, as it requires special
143+
# handling in Bazel.
144+
compiler_flag: "-g0"
145+
146+
# Conservative choice for -O
147+
# -O3 can increase binary size and even slow down the resulting binaries.
148+
# Profile first and / or use FDO if you need better performance than this.
149+
compiler_flag: "-O2"
150+
151+
# Disable assertions
152+
compiler_flag: "-DNDEBUG"
153+
154+
# Removal of unused code and data at link time (can this increase binary size in some cases?).
155+
compiler_flag: "-ffunction-sections"
156+
compiler_flag: "-fdata-sections"
157+
linker_flag: "-Wl,--gc-sections"
158+
}
159+
linking_mode_flags { mode: DYNAMIC }
160+
}
161+
162+
toolchain {
163+
abi_version: "local"
164+
abi_libc_version: "local"
165+
builtin_sysroot: ""
166+
compiler: "compiler"
167+
host_system_name: "local"
168+
needsPic: true
169+
target_libc: "macosx"
170+
target_cpu: "darwin"
171+
target_system_name: "local"
172+
toolchain_identifier: "local_darwin"
173+
174+
tool_path { name: "ar" path: "/usr/bin/libtool" }
175+
tool_path { name: "compat-ld" path: "/usr/bin/ld" }
176+
tool_path { name: "cpp" path: "/usr/bin/cpp" }
177+
tool_path { name: "dwp" path: "/usr/bin/dwp" }
178+
tool_path { name: "gcc" path: "clang/bin/crosstool_wrapper_driver_is_not_gcc" }
179+
cxx_flag: "-std=c++11"
180+
ar_flag: "-static"
181+
ar_flag: "-s"
182+
ar_flag: "-o"
183+
linker_flag: "-lc++"
184+
linker_flag: "-undefined"
185+
linker_flag: "dynamic_lookup"
186+
# TODO(ulfjack): This is wrong on so many levels. Figure out a way to auto-detect the proper
187+
# setting from the local compiler, and also how to make incremental builds correct.
188+
cxx_builtin_include_directory: "/"
189+
tool_path { name: "gcov" path: "/usr/bin/gcov" }
190+
tool_path { name: "ld" path: "/usr/bin/ld" }
191+
tool_path { name: "nm" path: "/usr/bin/nm" }
192+
tool_path { name: "objcopy" path: "/usr/bin/objcopy" }
193+
objcopy_embed_flag: "-I"
194+
objcopy_embed_flag: "binary"
195+
tool_path { name: "objdump" path: "/usr/bin/objdump" }
196+
tool_path { name: "strip" path: "/usr/bin/strip" }
197+
198+
# Anticipated future default.
199+
unfiltered_cxx_flag: "-no-canonical-prefixes"
200+
# Make C++ compilation deterministic. Use linkstamping instead of these
201+
# compiler symbols.
202+
unfiltered_cxx_flag: "-Wno-builtin-macro-redefined"
203+
unfiltered_cxx_flag: "-D__DATE__=\"redacted\""
204+
unfiltered_cxx_flag: "-D__TIMESTAMP__=\"redacted\""
205+
unfiltered_cxx_flag: "-D__TIME__=\"redacted\""
206+
207+
# Security hardening on by default.
208+
# Conservative choice; -D_FORTIFY_SOURCE=2 may be unsafe in some cases.
209+
compiler_flag: "-D_FORTIFY_SOURCE=1"
210+
compiler_flag: "-fstack-protector"
211+
212+
# Enable coloring even if there's no attached terminal. Bazel removes the
213+
# escape sequences if --nocolor is specified.
214+
compiler_flag: "-fcolor-diagnostics"
215+
216+
# All warnings are enabled. Maybe enable -Werror as well?
217+
compiler_flag: "-Wall"
218+
# Enable a few more warnings that aren't part of -Wall.
219+
compiler_flag: "-Wthread-safety"
220+
compiler_flag: "-Wself-assign"
221+
222+
# Keep stack frames for debugging, even in opt mode.
223+
compiler_flag: "-fno-omit-frame-pointer"
224+
225+
# Anticipated future default.
226+
linker_flag: "-no-canonical-prefixes"
227+
228+
# Include directory for cuda headers.
229+
cxx_builtin_include_directory: "/usr/local/cuda-8.0/include"
230+
231+
compilation_mode_flags {
232+
mode: DBG
233+
# Enable debug symbols.
234+
compiler_flag: "-g"
235+
}
236+
compilation_mode_flags {
237+
mode: OPT
238+
# No debug symbols.
239+
# Maybe we should enable https://gcc.gnu.org/wiki/DebugFission for opt or even generally?
240+
# However, that can't happen here, as it requires special handling in Bazel.
241+
compiler_flag: "-g0"
242+
243+
# Conservative choice for -O
244+
# -O3 can increase binary size and even slow down the resulting binaries.
245+
# Profile first and / or use FDO if you need better performance than this.
246+
compiler_flag: "-O2"
247+
248+
# Disable assertions
249+
compiler_flag: "-DNDEBUG"
250+
251+
# Removal of unused code and data at link time (can this increase binary size in some cases?).
252+
compiler_flag: "-ffunction-sections"
253+
compiler_flag: "-fdata-sections"
254+
}
255+
}

0 commit comments

Comments
 (0)