问题

报错:

ValueError: Argument must be a dense tensor: [array([[0., 0., 0., ..., 0., 0., 0.],
       [0., 0., 0., ..., 0., 0., 0.],
       [0., 0., 0., ..., 0., 0., 0.],
       ...,
       [0., 0., 0., ..., 0., 0., 0.],
       [0., 0., 0., ..., 0., 0., 0.],
       [0., 0., 0., ..., 0., 0., 0.]], dtype=float32)] - got shape [1, 10000, 10], but wanted [1].

原始代码

...
final_result = sess.run(fetches=[y], feed_dict={x: mnist.test.images})
...
y = tf.Variable(initial_value=final_result, name=TENSOR_NAME)
...

大概意思就是将一个神经网络的输出final_result作为参数, 传给下个方法, 下个方法以这个参数为初始值创建一个tf.Variable对象.

解决办法

...
final_result = sess.run(fetches=y, feed_dict={x: mnist.test.images})
...
y = tf.Variable(initial_value=final_result, name=TENSOR_NAME)
...

将sess.run中包裹y的方括号去除就可以了.

想了一下, 可能是因为传入run方法的参数fetches存在类型判断, 如果是list类型, 则返回结果也是值的list类型; 如果是单个操作, 则可能直接返回的是值.

    a = tf.get_variable(name='a', shape=(1,), initializer=tf.constant_initializer(value=0))
    b = tf.get_variable(name='b', shape=(1,), initializer=tf.constant_initializer(value=1))
    c = a + b
    d = tf.add(x=a, y=b, name='d')
    e = tf.get_variable(name='e', initializer=tf.truncated_normal(shape=(2, 3), stddev=1, mean=0))
    f = tf.get_variable(name='f', initializer=tf.truncated_normal(shape=(2, 3), stddev=1, mean=0))
    g = e + f
    h = tf.add(x=e, y=f, name='h')
    with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())
        print(a.name)  # a:0
        print(b.name)  # b:0
        print(c.name)  # add:0
        print(d.name)  # d:0
        print(e.name)  # e:0
        print(f.name)  # f:0
        print(sess.run(a))  # [0.]
        print(sess.run(b))  # [1.]
        print(sess.run(c))  # [1.]
        print(sess.run(d))  # [1.]
        print(sess.run(e))
        # [[-0.9891971   0.5030212  -0.48883826]
        #  [-1.4214745  -0.6322079  -0.4219446 ]]
        print(sess.run(f))
        # [[-0.10764185  0.6467568   0.03688372]
        #  [ 1.1124796  -0.89837056  0.67874044]]
        print(sess.run(g))
        # [[-1.096839    1.149778   -0.45195454]
        #  [-0.3089949  -1.5305784   0.25679585]]
        print(sess.run(h))
        # [[-1.096839    1.149778   -0.45195454]
        #  [-0.3089949  -1.5305784   0.25679585]]

问题描述

./EXAMPLE.py:407: UserWarning: An unusually high number of `Iterator.get_next()` calls was detected.
    This often indicates that `Iterator.get_next()` is being called inside a training loop, which will cause gradual slowdown and eventual resource exhaustion. 
    If this is the case, restructure your code to call `next_element = iterator.get_next()` once outside the loop, and use `next_element` as the input to some computation that is invoked inside the loop.
  warnings.warn(GET_NEXT_CALL_WARNING_MESSAGE)
libc++abi.dylib: terminating with uncaught exception of type std::__1::system_error: thread constructor failed: Resource temporarily unavailable

Process finished with exit code 134 (interrupted by signal 6: SIGABRT)

查看代码, 的确是将get_next放在了loop里面.

iterator = dataset.make_one_shot_iterator()
        while True:
            try:
                features, labels = iterator.get_next()
                features_val = sess.run(features)
                labels_val = sess.run(labels)
                features_val = np.array(features_val).reshape((32,28*28))
                labels_val = np.array(labels_val).reshape((32,1))
                sess.run(fetches=self.train_op, feed_dict={self.x: features_val, self.y: labels_val})

            except tf.errors.OutOfRangeError as _e:
                break

改正之后就正确了.

iterator = dataset.make_one_shot_iterator()
features, labels = iterator.get_next()
        while True:
            try:
                features_val = sess.run(features)
                labels_val = sess.run(labels)
                features_val = np.array(features_val).reshape((32,28*28))
                labels_val = np.array(labels_val).reshape((32,1))
                sess.run(fetches=self.train_op, feed_dict={self.x: features_val, self.y: labels_val})

            except tf.errors.OutOfRangeError as _e:
                break

Debug in TensorFlow

官方介绍Guide – Debugging – TensorFlow Debugger中已经介绍的很详细了, 但是有几个疏漏的点需要补充一下.

1. 在 使用 tfdbg 调试模型训练 章节中提到了可以通过自定义filter进行相关张量的筛选.

def my_filter_callable(datum, tensor):
  # A filter that detects zero-valued scalars.
  return len(tensor.shape) == 0 and tensor == 0.0

sess.add_tensor_filter('my_filter', my_filter_callable)

该filter是通过添加到Session中发挥作用的. 而Estimator则是在内部管理Session, 无法通过这种方式显式的增加filter.

# First, let your BUILD target depend on "//tensorflow/python/debug:debug_py"
# (You don't need to worry about the BUILD dependency if you are using a pip
#  install of open-source TensorFlow.)
from tensorflow.python import debug as tf_debug

# Create a LocalCLIDebugHook and use it as a monitor when calling fit().
hooks = [tf_debug.LocalCLIDebugHook()]

# To debug `train`:
classifier.train(input_fn, steps=1000,  hooks=hooks)

目前看起来正确的方式应该是通过LocalCLIDebugHook实例的add_tensor_filter方法进行添加.

sess通过tf_debug.LocalCLIDebugWrapperSession包裹之后, 其新添了add_tensor_filter方法, 源码是

  def add_tensor_filter(self, filter_name, tensor_filter):
    """Add a tensor filter.

    Args:
      filter_name: (`str`) name of the filter.
      tensor_filter: (`callable`) the filter callable. See the doc string of
        `DebugDumpDir.find()` for more details about its signature.
    """

    self._tensor_filters[filter_name] = tensor_filter

而LocalCLIDebugHook中的add_tensor_filter方法和上一种是相同的.

  def add_tensor_filter(self, filter_name, tensor_filter):
    """Add a tensor filter.

    See doc of `LocalCLIDebugWrapperSession.add_tensor_filter()` for details.
    Override default behavior to accommodate the possibility of this method being
    called prior to the initialization of the underlying
    `LocalCLIDebugWrapperSession` object.

    Args:
      filter_name: See doc of `LocalCLIDebugWrapperSession.add_tensor_filter()`
        for details.
      tensor_filter: See doc of
        `LocalCLIDebugWrapperSession.add_tensor_filter()` for details.
    """

    if self._session_wrapper:
      self._session_wrapper.add_tensor_filter(filter_name, tensor_filter)
    else:
      self._pending_tensor_filters[filter_name] = tensor_filter

报错tensorflow.python.training.basic_session_run_hooks.NanLossDuringTrainingError: NaN loss during training.

是在调用wide_deep的wide过程中报错的.
用tfdbg查看, 发现是在节点linear/linear_model/item_id_embedding/Reshape中存在全部为nan的张量.

 2 input(s):
      [Reshape] linear/linear_model/item_id_embedding/Reshape
      [Identity]

Tensorflow估计模型随损失= NaN而发散?
Deep-Learning Nan loss reasons
Tensorflow NaN loss during training: trying to reshape logits and labels